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Executive Summary 


The project 


ThinkForward is a coaching programme, developed by Impetus Private Equity Foundation, which is 
designed to support secondary school pupils. ThinkForward is targeted at pupils who have been 
identified as being at high risk of not being in education, employment or training (NEET) following the 
completion of compulsory education. Coaches are assigned to schools and work with selected pupils 
as they progress through Key Stage 4 (GCSEs), with the aim of supporting them to make a successful 
transition into adulthood. The programme provides targeted support tailored to pupils’ needs through 
one-to-one and group work. The programme usually works with young people aged 14 - 19, though 
this pilot focused only on the intervention up to age 16, prior to pupils leaving school. The pilot 
involved Year 10 and 11 pupils in four London secondary schools, beginning in January 2014. The 
Y11 pupils received up to six months of ThinkForward until their GCSEs in summer 2014. The Y10 
pupils received up to 18 months of ThinkForward up until their GCSEs in summer 2015. 


The pilot was evaluated using a range of qualitative methods and a small Randomised Controlled Trial 
(RCT). The aims of the evaluation were to determine whether a future large-scale evaluation was 
possible, and to gain an initial estimate of the programme’s impact on GCSE scores, the likelihood of 
continuing into post-compulsory education, and decreasing pupil absences. Randomisation took place 
at both school and pupil level. Within the two randomly assigned intervention schools, eligible pupils 
were randomly allocated either to an intervention group or a within-school control group. Across the 
four schools, there were 181 pupils in the Y11 cohort, 40 of whom received the intervention, and 160 
pupils in the Y10 cohort, 37 of whom received the intervention. 


Key conclusions 


1. The approach is not ready for a large school level randomised controlled trial. 


2. Before a further trial, it would be necessary to solve data collection issues and to develop an 
approach that allows the intervention to be targeted at pupils at risk of becoming NEET, while 
conducting a rigorous randomised controlled trial. 


. Future trials should not use pupil-level randomisation because it is likely that the intervention has 
‘spillover’ effects to other pupils. 


4. The trial did not find evidence that the programme had an impact on GCSE scores, absences, or 
attitudes. This was a small pilot RCT so these findings have low security. 


. Teachers reported benefits, particularly during the later months of the pilot, where they observed 


impacts on the intervention pupils’ behaviour. 


What are the findings? 


The study found a number of issues would need to be addressed before a future RCT is conducted: 
for example, further work is needed so that the intervention is able to accurately target pupils at risk of 
becoming NEET, while at the same time meeting the requirements of an RCT research design that 
pupils are randomly allocated to either receive the intervention or not. Another risk that would need to 
be mitigated is that some pupils who are not allocated to the intervention might benefit from the 
intervention, for example, if ThinkForward Coaches are asked to teach regular classes, as was 
reported in the process evaluation. This could result in the impact of the programme being 
underestimated and future RCTs would need to be designed to minimise this risk. Future trials should 
also employ approaches that might increase response rates to survey data, while making sure that 
any data is collected independently by researchers not involved in the delivery of the intervention. 
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The pilot found no positive impact on unauthorised absences or GCSE scores amongst students that 
took part in the programme. 


Attitudes to higher education, and the ‘ThinkForward Mindsets’ of ‘aspiration’ and ‘determination’ were 
measured using a survey. Responses showed a sharp decline in aspiration to attend higher education, 
measured in terms of expectations of applying to university, getting a place and graduating, and 
limited evidence of increases in ‘determination’ and ‘aspiration’, measured using survey questions. 
The survey results should however be treated with caution due to the low response rate within the pilot 


Evidence from the process evaluation suggests that ThinkForward is feasible to deliver in schools. 
School leads, coaches themselves and young people from intervention cohorts across both schools 
reported that they believed the programme was beneficial. Coaches and teachers felt that the support 
of a trusted adult help pupils when they were considering GCSE options in Year 9 and once they left 
school, as well as during Years 10 and 11. However, the complexity of options beyond school, and the 
numerous policies and procedures in place to support young people with behavioural and motivational 
issues, means it is difficult to evaluate interventions like ThinkForward in isolation from other 
influences. 


The costs of ThinkForward are high in comparison to other interventions, with a cost of £2426.50 per 
pupil per year during the pilot. Although, the costs of the intervention have since been reduced, and 
schools often do not pay for the entire intervention themselves, the high cost could make recruiting 
schools to a large school-level trial expensive. 


This evaluation was undertaken when ThinkForward was in a relatively early stage of its development, 
and the findings should be considered in this context. ThinkForward is committed to developing the 
programme and has already instituted many of the improvements that this report recommends. 


How was the pilot conducted? 


The pilot collected evidence for the ThinkForward programme across four outcome areas; two (KS4 
attainment and unauthorised absences) drew on data from the National Pupil Database (NPD);and 
two (future educational expectations, such as likelihood of going to university, and ThinkForward ‘mind 
sets’, such as aspiration and determination) drew on data collected through the longitudinal survey 
issued before the start of the trial, after the first six months of the trial and at the end of the trial. 


This was a small pilot RCT, so findings have low security. This means that the outcome data should 
be treated with caution. The pilot also suffered from low response rates to surveys and data collection 
problems. These risk introducing bias and limited the possible analysis on future education 
expectations and ThinkForward ‘mind sets’. 


Question Tare i ale] Comment 
Is there evidence of No! Evidence of impact was not found across any of the 
promise? Unclear four outcome areas. 
Was the approach Vda The process evaluation found positive comments from 
feasible? the students and staff involved in the approach. 
The pilot found methodological challenges around data 
Is the approach ready to be Not yet collection, and spillover, which make a within-school 


evaluated in a trial? trial unsuitable. The approach is not ready for a large 


school-level RCT. 
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1. Introduction 


1.1 Intervention 


ThinkForward (TF), developed by the Private Equity Foundation, is designed to provide highly trained 
coaches to work with disadvantaged young people from Y10 as they progress through to GCSEs and 
post-16 choices (including training and employment opportunities), Supporting them to make a 
successful transition into adulthood. 


Schools identify the young people who are most at risk of dropping out of education and employment. 
Coaches then develop relationships with those individuals, providing targeted support tailored to their 
needs in the form of both one-to-one and group work. 


This pilot followed two cohorts of young people in four London schools within the 2013/14 and 2014/15 
academic years. Cohort 1 were in Y11 at the start of the intervention and received up to six months of 
the ThinkForward coaching programme between October 2013 and the end of the 2013/14 academic 
year. Cohort 2 were in Y10 at the start of the trial and received up to 18 months of the ThinkForward 
coaching programme from October 2013 until the end of the 2014/15 academic year. 


The main aims of the pilot were methodological in nature, but a secondary aim was to examine 
evidence of impact for the ThinkForward coaching programme across four outcome areas: KS4/GCSE 
attainment, unauthorised absences, future educational expectations and two ThinkForward designed 
psychological mind set outcomes (‘aspiration’ and 'determination’). A complex research design was 
employed that involved randomisation at both the school and individual pupil level. Additionally, at 
baseline, detail on friendship and family links were collected in order to examine whether the 
measured impact of ThinkForward spilled over from the intervention to control group through these 
friends/family links. 


Alongside this collection of data that focused on impact and spill over, a process evaluation collected 
data from coaches, school leaders and participating pupils. 


1.2 Background evidence 


The theory of change underpinning ThinkForward is that by providing a stable relationship with an 
individual dedicated to thinking about all aspects of their future, young people can be guided down a 
path towards further education or training, and ultimately future employment. ThinkForward uses a 
coaching methodology to support young people to identify long term ambitions, short term goals 
towards these and then the immediate actions required to achieve these. 


Coaching and mentoring are not entirely analogous, but there is limited evidence on the impact of 
coaching interventions for young people. Evidence on the impact of mentoring and coaching 
interventions of this type is mixed; for example there is often an immediate improvement in behaviour, 
engagement and attainment at the outset of such intensive supportive relationships with ‘non-teaching’ 
staff which may not always bear longer term improvement. In such cases this is said to be the result of 
young people being in receipt of levels of attention they are unused to, with the effect dissipating as 
they become used to the attention to their needs (Franke and Kaul, 1978). There is evidence from the 
US, and in particular with at-risk groups, that ‘natural non-parental mentors’ within the specific 
community can have a longer, ongoing beneficial effect on young people's outcomes: Rhodes et al 
(1992, and 1994) noted that several investigators studying adolescent resiliency have found that non- 
parental adults frequently have a positive effect by providing support to at-risk youth (Zimmerman et 
al, 2002, p.243). It should be noted that ThinkForward is more structured than many of those 
highlighted in much of the literature, and uses trained coaches rather than volunteer mentors. 
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While these investigations suggest that educational outcomes can be positively affected in the longer 
term, they are primarily focused on community based and natural mentoring relationships. 
ThinkForward is specifically designed to create similar relationships within educational settings (with 
the addition of significant out-of-school and post-16 contact) and in such contexts the findings are 
more mixed. Rodriguez-Planas (2010) reported on the first randomised trial in the US to analyse the 
short and long-term educational and employment impacts of an after-school programme that offered 
disadvantaged high-school youth: mentoring, educational services, and financial rewards with the 
objective to improve high-school graduation and post-secondary schooling enrolment. Here the 
findings were found to be relatively short-lived as other factors relating to the motivation of the 
individual eventually returned to the fore.: 


Average impacts reveal that the hefty beneficial educational outcomes quickly faded away. 
Heterogeneity matters. While encouraging results are found for the younger youth (improved high- 
school graduation and post-secondary schooling enrolment for males); detrimental long-lived 
outcomes suggest that extrinsic rewards may be crowding out intrinsic motivation. (Rodriguez-Planas, 
2010). 


Reid (2002) and Tucker (2013) offer evidence from the increasing use of mentoring and coaching 
programmes in the UK context, specifically as exemplars of New Labour ‘school improvement' policies 
including those associated with Connexions and Education Action Zones. Reid found that to be 
successful in educational settings, tasks required of the young people would need to be clearly 
defined and realistic with attainable (and measurable) goals. Mentors require appropriate and relevant 
knowledge and experience of both life and learning: When both pupils and mentor alike enjoy the 
companionship provided by a tutoring relationship, the potential for learning is usually greater (Reid 
2002, 158). Tucker emphasised the ‘whole school’ effect, combining pastoral support workers efforts 
often in terms of multiple interventions with overlapping cohorts of young people: 


It is at the level of the individually focused intervention that the most specialised forms of work with 
young people are being carried out. At the same time, it is interesting to note how individual and group 
activities are frequently combined in developing ‘packages of pastoral care’. (Tucker, 2013, 286). 
Zimmerman et al (2002) looked at the effects of several studies of non-parental adult mentors, using 
the resiliency theory framework. These so-called, ‘natural mentors' were reported to have 
‘compensatory effects on problem behaviors... and on school attitudes' (Zimmerman et al 2002, 221). 
They found that '....mentors tended to perform teaching, challenging, and role modelling functions’ 
(Ibid, 222). They noted that one-to-one interactions focused on discussing personal and intellectual 
matters were the most common activities reported as successful in this context. 


Tucker also reported on the individual interventions which are employed to explore the underlying 
causes of poor behaviour (Such as those that ThinkForward focuses on) including, insecurity, social 
isolation and resultant feelings of anger that can derail educational focus. This work suggests that 
educational psychological support and therapeutic counselling can be effective, especially when 
combined with focused pastoral responses that went beyond the boundaries of the school to include 
parents and carers (Tucker, 2013, 287). ThinkForward also has a beyond-school and interaction with 
parents and carers element, although, as our evaluation is of a time-limited pilot trial version we 
should not expect the same degree of successful outcomes. 


Gorard (2012) was among the first in the UK to explore causal links between mentoring and attitudinal 
change that, in turn, potentially impact educational attainment. Employing a four stage model of 
causation - association, sequence, intervention, and explanation - he found that while some mental 
concepts such as external motivation "showed promise" in raising educational aspirations: there was 
no clear evidence that intervening to change the educational attitudes of disadvantaged students will 
lead to enhanced attainment. Given that structural inequalities are the main determinant of attainment, 


Education Endowment Foundation 7 


ThinkForward 


for Gorard, improved attitudes towards education alone are unlikely to demonstrate much 
improvement without the competence to change the real, systemic, causes (Gorard, 2012). 


However, Higgins et al (2013) in their review of the evidence for the Sutton Trust/EEF toolkit, did 
identify conditions under which such interventions can be effective (Bernstein et al, 2009; Nunez et al, 
2013). They found that mentoring / coaching interventions have been seen to work with young people 
from disadvantaged backgrounds, where double the (average) impact on attainment has been 
reported compared to non-disadvantaged groups (Higgins et al, 2013). Benefits were found to include 
attitude to school, attendance and behaviour (which are the main foci of the ThinkForward 
programme). However, as with the US evidence noted earlier, school-based mentoring was found to 
be on average less effective than community based mentoring. This can be attributed to school-based 
versions offering less opportunity to develop lasting, trusting relationships with adult role models. 


Here the implication is that mentors are seen to be part of the school structure after a while; the initial 
and often personalised attention from ‘non-teaching’ adults works as long as the mentor is not seen as 
part of the educational system they are disaffected from. However, there is some support in the 
evidence (Higgins et al, 2013, cite a decade of consistent US findings) that programmes that have a 
clear structure and expectations, provide training and support for mentors, and use mentors from a 
professional background, are associated with more successful outcomes, and that such programmes 
are relatively inexpensive once training costs of the coaches is taken into account. 


1.3 Evaluation objectives 
As specified in the research protocol1, the pilot trial had two main aims: 


e To assess the feasibility of pupil-level randomisation in evaluating a pupil coaching / mentoring 
programme. 
e To inform the design of a larger scale RCT evaluation of ThinkForward 


In addition to these two main methodological aims, the pilot trial also aimed to identify whether the 
ThinkForward programme showed ‘promise’ in terms of having an impact on a range of pupil 
outcomes including attainment. 


Three research questions were developed: 


e How feasible is pupil-level randomisation in evaluating ThinkForward? 

e What is the impact of the ThinkForward coaching / mentoring programme on a set of 
measurable outcomes including attainment? 

e How does the ThinkForward coaching / mentoring programme operate to produce these 
outcomes? 


1.4 Pilot evaluation team 


e Sean Demack was the methodological director for the pilot and led on the quantitative design, 
analyses and reporting. 

e Colin McCaig was the subject specific director and led on the process evaluation and locating 
the evaluation within the research literature 

e Claire Wolstenholme was the project manager and was also involved in the process 
evaluation interviews, analyses and reporting 


1 See https://educationendowmentfoundation.org.uk/evaluation/projects/think-forward 
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e Laura Fumagalli assisted with the initial trial methodology, developed the model specification 
and advised on the quantitative design, analyses and reporting 


1.5 Ethical review 


The research design, data collection instruments and two forms of opt-out consent were approved by 
the Sheffield Hallam University (SHU) ethics committee in June 2013. 


The opt-out consent form for participation in the survey and to link survey and NPD data can be found 
in Appendix 1. This form was given out to Y10 and Y11 pupils within the four London schools. 


Following school-level randomisation, further consent was sought from Y10 and Y11 pupils within the 
two selected intervention schools. This was an opt-out consent form for participation in the RCT and 
the form used can be found in Appendix 2. 
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2. Methods 


2.1 Evaluation Design 


The pilot trial involved two cohorts of pupils. Cohort 1 was in Y11 at the start of the trial and took KS4 
(GCSEs) in summer 2014. Cohort 2 was in Y10 at the start of the trial and took KS4 in summer 2015. 
Therefore, cohort 1 experienced around six months and cohort 2 experienced around 18 months of 
the ThinkForward programme. 


Four secondary schools were involved in the pilot. The number of schools was specified by EEF within 
the evaluation contract and the pilot did not aim to be a fully powered trial. The main aims of this pilot 
are methodological. Whilst ‘evidence of promise’ is an important aspect, caution is advised when 
drawing conclusions from any single outcome. Evidence of promise across a range of items provides 
stronger evidence but a larger scale efficacy trial would be required before any causal conclusions 
could be made. 


A baseline survey was conducted in October 2013 for all those pupils who had given consent. The 
baseline survey collected data on: ThinkForward mind sets, expectations on the expected likelihood of 
continuing and succeeding into post compulsory education (including university) and friendship and 
family networks. Follow up surveys took place in June 2014 (survey 2 for cohorts 1 and 2) and May 
2015 (survey 3 for cohort 2 only). Alongside the survey data, data from the National Pupil Database 
(NPD) was obtained at baseline, at the end of the first academic year (2013/14) and at the end of the 
final academic year (2014/15) of the pilot. 


Baseline data (on respondents’ characteristics and potential outcomes) are useful to check the 
balance and to adjust for pre-intervention differences between the intervention and the control 
groups. . Failing to take into account pre-intervention differences between the intervention and the 
control group can result in biased estimates of the program’s effects. The issue of the baseline 
imbalance and our approach to limiting potential bias in the estimation of program’s effects is 
addressed section 2.4. The baseline survey also collected social network data. Specifically, the survey 
included two items asking respondents for the names, genders, ages and year-group of up to five 
friends and family members in Y10 and Y11. The purpose of collecting information on friend/family 
networks prior to the intervention was to explore the existence and nature of spillover effects within the 
two intervention schools. 


The pilot aimed to provide evidence to inform the choice of the counterfactual (control group). In 
particular, the trial was set up such that two types of control groups were tested: a “pupil-level within- 
school control group”, where intervention and control individuals belong to the same schools, and a 
“school-level control group”, where intervention and control group individuals belong to different 
schools. 


The advantages of a within-school design are: 1) comparability: A within-school control group is likely 
to be more similar to the intervention group than a school-level control group 2) sample size: A within- 
school control group is likely to require a smaller sample size than a school-level control group for a 
given statistical power 3) ease of recruiting schools: Schools where at least some pupils are given the 
intervention may be more willing to participate in the study than schools which serve as a pure control 


group. 


The main disadvantage of the pupil-level control group over the school-level control group is the 
potential contamination due to spillover effects. If the intervention also affects the within-schools 
control group through spillover effects, this will result in inaccuracy in estimating the programme’s 
impact. Spillover effects could arise through interactions between participants or through coaches 
delivering the programme to control group pupils. 
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2.2 Intervention 


The ThinkForward intervention involves placing a trained coach within a school in order to work with 
pupils from age 14 onwards who were 'most at risk of becoming NEET' (Not in Education, Employment 
or Training?). Coaches are employed full time and should develop a relationship with pupils in order to 
address all aspects of their school and home lives. Coaches should have had (and further develop 
through ongoing training) good knowledge of opportunities open to young people in order to signpost 
pupils to opportunities outside of the school. This is to enable a smooth transition to Further Education 
(FE), employment, or vocational training. Coaches work with pupils on both a one to one basis and in 
group sessions in order to: 


e = Build life skills and confidence 

e Provide access to employers and the world of work 
e Connect youth to relevant services and networks 

e Help transition from school to work 

e Support through challenges at home and school 


ThinkForward’s ultimate outcome is to support young people to progress into sustained employment 
or training after they graduate from the programme. 


Coaches follow a four step process in order to carry out their work with pupils: 


Identify participants 


At this stage pupils are scored according to a number of factors which should help to identify their risk 
of becoming NEET. Baseline information is collected and put into a scoring mechanism; the system 
then automatically runs a selection panel. The selection panel is a manual process that ThinkForward 
use to ensure that all data is complete and then to confirm pupils for enrolment to the ThinkForward 
programme. 


ThinkForward works with young people predicted to be most at risk of unemployment when they leave 
school. They have developed a bespoke ‘risk of NEET’ scoring mechanism in order to identify these 
pupils, which includes known risk factors such as poor attendance and behaviour at school, below 
expected academic progress and Special Educational Needs status. Data is provided by partner 
schools which allows ThinkForward to identify a shortlist of young people to enroll. ThinkForward 
convenes a school panel including staff such as heads of year and pastoral support team to have a 
more detailed discussion to decide a coaches’ cohort from the shortlist. For the pilot, ThinkForward 
provided predicted NEET risk scores for Y10 and Y11 pupils within the four pilot schools in excel 
format. These predicted NEET risk scores were used to identify eligible pupils and for the within- 
school, pupil-level randomisation (See section 2.3).Understand participants’ needs 


Coaches have a range of information about participants’ baseline situation from the identification 
process. They also continue to actively engage with school staff to find out more details about young 
people’s educational and social history, for example whether they are involved with the criminal justice 
system or have a family history of unemployment. 


Pupils are scored against the ThinkForward’s Personal Development tracker, a bespoke tool created 
by ThinkFoward in partnership with behavioural assessment consultants a&dc. This involves young 
people undertaking an online psychometric test (on a platform called Apollo) from which they and their 
Coach receive a Personal Development report highlighting areas of strength and_ potential 
improvement. 


2 See http:/www.thinkforward.org.uk/about/ for more on the ThinkForward coaching programme 
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Scores are summed across scales and are then compared to a norm group. On this basis, Standard 
Ten (Sten) scores are generated. A participant's scores can be viewed either as a report which is 
generated through the platform, or as an excel spreadsheet which can be downloaded from Apollo. 


Information collected from the identification process, school and the Apollo data for each pupil is 
recorded on a central management information system. Coaches are typically asked to select 15 
pupils per year group who are identified as needing intensive support and 20 needing what is referred 
to as standard support. Pupils are then assessed by coaches in person on a series of competencies 
such as decision making ability. Coaches use a standard enrolment form to understand participants 
needs, with pupils encouraged to discuss future career options, what they might need to get there and 
explore additional potential barriers including in areas such as drug/alcohol issues, family and housing 
which were not revealed in the identification process. 


After selection, coaches make contact with pupils, in the first instance through letters to the pupils and 
their parents. Coaches then approach young people for introductions. At the initial meetings coaches 
complete an Initial Action Plan with the pupil, this outlines expectations on behalf of the pupil and 
coach, and both parties form an agreement. An enrolment form is also completed with the pupil which 
gathers background information. Lastly a consent form is completed with the pupil once the coach 
feels the pupil is willing to be enrolled in the programme. 


Design an action plan 


The initial action plan is used as a starting point, and then a detailed action plan is downloaded and 
used. Specific, Measurable, Assignable, Realistic and Time related (SMART) targets are developed 
with pupils. ThinkForward uses the GROW coaching model- (Goal, Reality, Opportunities, What next?) 
in order to develop plans for pupils. These goals are reviewed typically once per term and new ones 
developed where necessary. 


Do and review 


Coaches hold a mix of one to one and group sessions with pupils (the ratio of these will vary 
depending on what is deemed to be most effective for the coach and pupils) where plans and progress 
are reviewed. Coaches typically see pupils approximately twice per half term for a planned, structured 
session lasting around 45 minutes as well as informal ‘check ups’ in the interim. Group work sessions 
typically take place twice per half term delivered by the coach, and last approximately 30-45 minutes. 
The frequency of one to one and group work sessions and ratio between these can vary depending on 
what is deemed to be most effective for the coach and pupils. The content of the one to one and group 
sessions are tailored to the needs of the group based on needs identified through the baseline needs 
assessment and Personal Development tracker. All sessions are recorded on a management 
information system. Coaches can draw on an intervention catalogue in order to offer available 
services, activities or experiences to pupils. 


In late year 10/early year 11 pupils also undertake a series of 6-8 business mentoring sessions where 
they are paired with a mentor from one of ThinkForward's corporate partners. These sessions are 
designed to give pupils an insight into the world of work, an exploration of the roles and sectors which 
might be suitable for their future employment and equipping them with the necessary resources and 
experience to secure a Suitable job once they leave school. The latter includes support with CV and 
application writing and interview practice. All sessions are recorded on a management information 
system. Coaches can draw on an intervention catalogue in order to offer available services, activities 
or experiences to pupils. 


Coaches are tasked with producing two pupil case studies per year which detail the progress the pupil 
has made during their time with ThinkForward. The case studies have 6 questions which are designed 
to demonstrate the impact that ThinkForward has had on the pupil's life. 
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In addition to working with pupils, coaches have termly review meetings with the school lead. 
Evidence of outcomes are then entered onto the management information system recording progress 
against the scoring mechanism (based on mind-sets and employability skills). 


This evaluation was undertaken when ThinkForward was in a relatively early stage of its development, 
and the findings should be considered in this context. ThinkForward is committed to developing the 
programme and has already instituted many of the improvements that this report recommends. 


2.3 Sampling and recruitment 


Four London secondary schools were recruited by ThinkForward between April and November 2013°. 
By signing the memorandum of Understanding (MoU%), the school leaders committed to facilitate the 
data collection over the trial period (January 2014 to June 2015). 


Data was collected on 22 schools in the three London boroughs where ThinkForward operates 
covering school size, whether or not they had a sixth form and percentage of pupils with 
characteristics which are predictive of academic attainment or progression (eligibility for Free School 
Meals, first language not being English, GCSE attainment). This information was used as the basis for 
pairing schools with similar profile to be part of the study. 


ThinkForward provided some detail on the difficulties they faced in recruiting the four schools. They 
reported that recruitment was challenging for several reasons. The foremost of these was that schools 
were asked to sign up in principle, without knowing whether or not they would receive the intervention. 
Schools that were keen to welcome ThinkForward were hesitant to effectively bar themselves should 
they have been chosen as a control school; persuading those who were less far along the journey was 
also made harder by the uncertainty. Secondly, identifying pairs of schools with suitably matched 
characteristics, particularly due to some schools having in house sixth forms which was identified as 
being an important factor, as well as similar levels of the other factors for the control group to be 
meaningful. Finally, there were a number of practical issues including needing to get matched pairs of 
schools to commit to the same start date for the project and the fact that there was also a certain 
amount of extra admin conferred by the rigour of the university's methods. The incentive payment 
went some way to helping with this, but ensuring they signed up fully to the details often took several 
meetings with busy deputy heads. 


Following the signing of the MoU, an opt-out consent form was sent to all of pupils in Y10 and Y11 of 
the four participating schools. Consent was sought to participate in the surveys and to link individual 
survey data to data from the National Pupil Database (NPD)°. A two week period was allowed to 
elapse for opt-out forms to be returned. 


The school level randomisation took place in November 2013. The four schools were stratified into two 
groups: the first included two schools that offered KS5 via a sixth form and the second included two 
schools without a sixth form. Through a public coin toss, which took place on November 11th 2013, 
one of the two schools with 6th forms was selected to become the first intervention school (where the 
intervention was offered to a randomly selected set of students) and the other school became part the 
school-level control group (where the intervention was not offered). Once the remaining two schools 
without sixth forms had also completed the baseline data collection, a second public coin toss 
determined which of these would be an intervention and control school. The school level 
randomisation was completed on November 28th, 2013. 


3 The pilot was originally due to commence from September 2013 but the difficulties in recruiting schools resulted 
in delaying the start of the trial and curtailing the time available for the pilot trial. 

4 See Appendix 4 for a copy of the schools Memorandum of Understanding (MoU) 

5 See Appendix 1 for the first (survey data collection & linkage to NPD) opt-out consent form. 
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A two week period between the school level and pupil level randomisation was allowed to seek pupil / 
parent opt-out consent for participation in the trial®. 


ThinkForward uses a scoring mechanism to identify young people in each schools who are predicted 
to be most at risk of becoming NEET when they leave school. These predicted NEET risk scores were 
used to identify eligible pupils and for the pupil-level randomisation. 


This pupil-level randomisation was done in three stages: 


e Stage one: a NEET-risk score for each student in three of the four schools’ was provided by 
ThinkForward for all Y10 and Y11 pupils. The NEET-risk score is a proxy for the subject- 
specific risk of being NEET (Not in Employment, Education or Training) after leaving school 
age. 

e Stage two: the median value of the NEET-risk score across the three schools was generated. 
This was equal to 32.2%. 

e Stage three: the pool of students eligible to receive of the programme was restricted to those 
with a NEET-risk above 32.2%. The 32.2% threshold was selected because it ensured that the 
minimum number or pupils within a year group was at least 40 pupils (20 of whom would be 
randomly selected to receive the ThinkForward programme). This compares with a median 
NEET risk score of 23.6% for ThinkForward’s other schools, giving an indication that pupils in 
the schools part of this study had a greater overall risk of NEET. 


ThinkForward wanted to ensure that they worked with at least some pupils with very high NEET-risk 
scores, as this was their target population. To try to ensure this, the initial group of eligible students 
with a NEET score higher than 32.2% was split further into two sub-groups: a) those with a NEET-risk 
score above the upper-quartile (i.e., those with NEET-risk score higher than 45.4%) b) those with 
NEET-risk score between 32.2% and 45.4%. The pupil-level randomisation then took place. First, 
pupils with a NEET risk score higher than 45.4% were randomly placed into the intervention or pupil- 
level control group using the SPSS software package. After this, pupils with a NEET-risk score 
between 32.2% and 45.4% were randomly selected to join the intervention group until this reached the 
limit of 20 pupils per year group. The remaining unselected pupils were placed into the pupil-level 
control group. 


The approach to randomisation ensured that pupils who opted out were not included into the pupil 
level randomisation. Whilst the pilot did suffer from notable issues of non-response for the follow up 
surveys, no pupils/parents opted out following the start of the trial in January 2014. However, following 
pupil level randomisation in December 2013, two cohort 2 (Y10) intervention group pupils were 
permanently excluded from school and so not included in the analyses. 


Another factor to note is that the trial was due to commence in September 2013 but in reality did not 
begin until January 2014 because of school recruitment difficulties. January 2014 was when coaches 
began to work with their randomly selected pupils within the two randomly selected intervention 
schools. However, following school level randomisation in November 2013, coaches worked within the 
four schools to gather data in order to calculate the predicted NEET risk score used to identify eligible 
Y10 and Y11 pupils. One of the coaches left ThinkForward in December 2013 and it was not until 
February 2014 before a new coach was in post. This resulted in a further one month delay to the start 
of the trial for one of the intervention schools. 


& See Appendix 2 for the second (RCT) opt out consent form. 

7 At the point of pupil level randomisation, ThinkForward could only provide data for pupils in three of the four 
schools. Predicted NEET risk scores were available for Y10 and Y11 pupils in both intervention schools but only 
one of the two control schools. Following the delayed start of the trial, after the school level randomisation was 
completed ThinkForward were keen to identify the Y10 and Y11 pupils that their coaches would begin working 
with in January 2014. For this reason, pupil-level randomisation drew on data from only three of the four schools. 
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Figure 2.3.1 presents a flow chart for the school and pupil level randomisation. 
Figure 2.3.1: Pilot evaluation of ThinkForward: Stages of school and pupil level randomisation. 


4 London Secondary Schools recruited. 2 had sixth forms, 2 did not. First Opt-out consent (for survey 
Recruitment (April - Oct data collection & NPD linkage) sent out to all Y10 & ¥11 pupils (process facilitated by ThinkForward 
2013) coaches). 


Evaluators: baseline Survey & NPD request (all YL0 & Y11 pupils who did not opt out in all four 


schools} 
Baseline Data Collection 
Oct- Nov 2013. 
{ . ) ThinkForward: gathered data in the four schools to calculate predicted NEET risk score for YL0 & Y11 
pupils in all four schools} 
School level Four schools placed into two groups (with & without sixth form). One school from each group was 
randomisation randomly selected to become an intervention school (two coin tosses) 


(Nov 2013) 


Two intervention schools. Two control schools 


Opt-out consent form 2 (RCT participation) 
sent out. 


Identification of eligible pupils Using a combined data set of Y10 and Y11 pupils from both intervention schools and one of the two 

(Dec 2013) control schools, the median predicted NEET risk score was calculated as 32.2%. Pupils with a score 
above 32.2% were identified as being eligible for ThinkF onvard. This criterion was applied to the final 
control school once the NEET risk data was provided by ThinkForward (in May 2014). 


Two intervention schools. Two control schools 
Totals: 207 YL0 & 239 Y11 pupils Totals: 308 Y10 & 315 Y11 pupils 
Eligible: 120 YL0 & 138 Y11 pupils Eligible: 113¥10 & 121 Y11 pupils 


Removal of pupils who did not Pupils who did not complete a baseline survey (either because they opted out formally viathe consent form or 
respond to baseline survey. (Dec informally by not completing the baseline survey) were then dropped from the sample of eligible pupils. 
2013) 


Two intervention schools. 
Eligible pupils with baseline survey 


95Y10& 115 Y11 pupils 
Within school (pupil level) 


randomisation using SPSS (Dec 
2013) 
Intervention group 
40 Y10 & 40 Y11 pupils 


2 Y10 pupils permanently 
excluded (Dec 2013) School level control 
Eligible pupils with baseline survey 


Final baseline samples (Dec Intervention group Pupil level control response 


2013) 38 Y10 & 40 Y11 pupils 5S Y10 & 75 Y11 pupils 69 YLO & 66 Y11 pupils 


Follow up data collection 
GCSE attainment obtained from NPD for 2013/14 (Cohort 1) and 2014/15 (Cohort 2). For Cohort 2, two cases were missing GCSE attainment data in 2014/15. 
Unauthorised absence data obtained from NPD for 2013/14 (Cohorts 1 and 2) and 2014/15 (Cohort 2). For Cohort 2, eight cases were missing unauthorised 


absence data for 2014/15. 
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This randomisation approach did ensure that ThinkForward coaches would be working with some 
pupils with a relatively high predicted NEET-risk score (i.e. above 45.4%). However, the approach also 
resulted in a baseline imbalance. The reason for this relates to the how the number of pupils receiving 
ThinkForward per year group / school was fixed whilst the number of pupils identified as ‘eligible’ was 
not. The intervention and control group samples were not balanced with respect to the NEET-risk 
score; the (smaller) intervention group had a higher mean NEET-risk score compared with the (larger) 
control group. A baseline imbalance was also observed with KS3 attainment, FSM, gender and SEN 
but not of the same magnitude as seen with the predicted NEET risk score. Table 2.3.1 below 
summarises the distributions of the predicted NEET risk score, KS3 attainment, %FSM, gender (% 
female) and SEN (% statement or school action plus statement) baseline variables. 


Table 2.3.1: Examination of baseline balance across treatment groups 
Mean predicted NEET risk score, mean KS3 attainment, %FSM, 
% Female and % SEN/School Action plus 


NEET risk score KS3 attainment* FSM Female SEN / 
SA+ 
| n= mean (sd) n= mean (sd) % % % 
Cohort 1 
47.5 . é é 
Total Sample 181 (12.65) 171 4.8 (0.88) 83% 39% 49% 
: 55.0 5 - é 
Intervention group 40 (14.00)) 37 4.7 (0.98) 80% 25% 40% 
Control group (total) 141 nies 134 | 4.9 (0.85) 84% 43% 51% 
Pupil level control 75 aoe 68 4.9 (1.01) 84% 40% 35% 
(11.57) 
School level control 66 te 66 4.8 (0.64) 83% 47% 70% 
Cohort 2 
46.7 a 5 ' 
Total Sample 162 (11.63) 162 4.9 (0.74) 83% 36% 44% 
Intervention group 51.4 
. 0, 0 10) 
38 (12.84) 38 4.8 (0.79) 79% 34% 47% 
Control group (total) 124 iee5 124 | 4.9 (0.73) 84% 37% 43% 
46.4 ' f : 
Pupil level control 55 (10.86) 55 5.1 (0.74) 85% 38% 27% 
School level control. «69 Pen 69  4.8(0.70) 83% 36% 55% 


* KS3 attainment was derived using the mean KS3 level in English, Mathematics & Science (teacher assessments) 


In short, our attempt to standardise the ThinkForward eligibility criteria coupled with the fixed number 
of intervention group participants and the desire for ThinkForward to ensure that their programme 
involved pupils with a high NEET risk score led to a baseline imbalance. Whilst this imbalance is most 
clearly seen with the predicted NEET risk score (intervention group pupils having a higher mean 
predicted NEET risk compared with the control), imbalance was also observed with respect to: 
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e KS3 attainment (lower on average for intervention group pupils compared with the control) 

e FSM (intervention group less likely to be classed as FSM compared with the control) 

e Gender (intervention group less likely to be female compared with the control) 

e = SEN (intervention group more likely with the pupil-level control and less likely than the school- 
level control to have a statement or school action plus SEN). 


A key issue that will need to be addressed in future trial designs is the development of eligibility criteria 
that are suited to an RCT design whilst also fitting the needs of the (coaching / mentoring) programme 
in terms of ensuring that the programme reaches the pupils it is aimed at needs to be developed. The 
criteria also need to take account of the fixed numbers of pupils per year group / school that will 
receive the programme. Our reflections on this can be found in section 4.8. Our approach to 
addressing potential estimation bias resulting from the baseline imbalance was to adopt a standard 
and difference in difference regression analytical approach and to include predicted NEET risk score, 
KS3 attainment, FSM, gender and SEN as explanatory variables within these models as specified in 
section 2.4.6 below. 


Pupil-level randomisation was completed in December 2013 after allowing for a second two-week opt- 
out consent period to elapse. Opt-out consent for participation in the RCT was sought from eligible 
pupils within the two intervention schools. The school and pupil-level randomisation took place 
between November 11th and December 13th 2013 and the public registration of the trial was 
confirmed on December 27th 2013. 


2.4 Outcomes and evidence of promise 


The pilot looked for evidence of promise across a range of outcomes that were drawn from the three 
NPD data requests and data from the (three-wave) longitudinal survey. Four groups of outcome 
variables were analysed. 


The first group captures Key Stage 4 (KS4) attainment, i.e. attainment in the General Certificate of 
Secondary Education (GCSE). KS4/GCSE Attainment is our primary outcome as specified by EEF in 
the original research specification. 


The remaining three groups of variables measure secondary outcomes, namely unauthorised 
absences, ThinkForward mind sets and expectations on the perceived likelihood of continuing into 
post compulsory education (including higher education). 


A pupil's history of unauthorised absence is part of the ThinkForward NEET risk scoring mechanism 
used to identify whether a pupil is likely to benefit from ThinkForward. Reduced numbers of 
unauthorised absence are an indication of an increase in educational engagement and for this reason 
this measure is included as a within the secondary outcomes for this pilot trial evaluation. 


The two mind set outcomes (‘aspiration’ and 'determination’) were selected by ThinkForward because 
they believed that they were the most likely to capture a positive impact from the programme within 
the pilot trial period. Measuring educational expectations is important to shed light on reasons why 
students from a low Socio Economic Status are underrepresented in post-compulsory education, and 
especially in higher education. 


This section introduces each of the four outcome areas and variables used to measure them. The 
section also notes any problems encountered in the data collection. As explained in section 2.4.3 
below, the problems relating to data from the longitudinal survey meant that the impact analyses of 
these outcomes could only be descriptive and causation cannot be attributable. Specifically, because 
of large problems of non-response and other data collection problems, analyses of the impact of 
ThinkForward on future educational expectations and ThinkForward mind set of participating pupils 
are limited to descriptive bivariate tables. 
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NPD data does not suffer the same non-response and data collection issues as the longitudinal survey 
and these variables were included into the regression analyses specified in section 2.4.6. This meant 
that for these analyses, the estimated impact of ThinkForward took account of clustering and included 
explanatory variables to address the baseline imbalance shown in section 2.3. For KS4/GCSE 
attainment, a series of standard linear regression models are specified. For unauthorised absences, a 
series of difference in difference linear regression models are specified. 


The standard and difference in difference regression analyses also examine the potential impact of 
spill over through the inclusion of a dummy variable that identified whether a pupil in the control group 
had a friend or family member in the intervention group (=1) or not (=0). The friends/family links 
dummy variable is introduced in section 2.4.7. The use of the friends/family links dummy variable 
within the regression analyses is also discussed. 


2.4.1 Primary Outcome - KS4 / GCSE attainment (NPD) 


Data on KS4/GCSE attainment were obtained from the National Pupil Database (NPD), an 
administrative dataset covering all students in state schools in England. For this reason, attainment 
data do not suffer from miss-reporting or non-response error/bias to the same extent as survey data 
might. Missing data can still pose a problem, but as noted below this was fairly minimal for this pilot. 
Opt-out consent to draw on NPD data and to link this to data was collected from the surveys was 
obtained prior to randomisation. For cohort 1, KS4/GCSE data for all participants who completed the 
baseline survey was requested in November 2014. A complete set of KS4/GCSE data was provided 
for the 181 participants in cohort 1 who completed Y11 in summer 2014. For cohort 2, KS4/GCSE 
data for all participants who completed the baseline survey was requested in December 2015. NPD 
KS4/GCSE data was provided for 160 of the 162 cases. One of the cases with missing data was 
located within the intervention group and the other missing case was in the school-level control group. 


To measure KS4 / GCSE attainment we used a continuous scale variable that captured overall 
attainment. Specifically, the KS4 / GCSE outcome variable used was the mean KS4 points score per 
GCSE (or KS4 equivalent) entry in 2014 (for cohort 1) and in 2015 (for cohort 2). 


Table 2.4.1 summarises the KS4/GCSE attainment variable for each cohort (cohort 1 and cohort 2) 
and each treatment groups (the intervention group, the pupil-level control group and the school-level 
control group). Summaries of two versions of the KS4/GCSE attainment variable are shown, firstly for 
all cases and secondly excluding low outliers. Table 2.4.1 presents summaries of the original variables 
that included low outlying cases alongside summaries of the variables when the low outlying cases are 
excluded 
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Table 2.4.1: Primary Outcome Variable: Key Stage 4 (GCSE) attainment (NPD). 
Mean KS4/GCSE points per KS4 exam 
Original variables & excluding low outlying cases 


Cohort 1 (KS4/GCSEs in 2014) 


Original Variable Excluding 2 low outliers 
n= Hat=rc lam (1e)) n= mean (sd) 
Total Sample 181 34.6 (8.54) 179 35.0 (7.76) 
Intervention group 40 35.4 (6.76) 40 35.4 (6.76) 
Control (Total) 141 34.4 (8.99) 139 34.9 (8.04) 
Pupil level control group 75 36.9 (10.13) 73 38.0 (8.15) 
School level control group 66 31.4 (6.41) 66 31.4 (6.41) 


Cohort 2 (KS4/GCSEs in 2015) 


Original Variable Excluding 11 low outliers 
n= Hat=r- 1am (10) n= Hat=r- 1am (Xe) 
Total Sample 160 31.5 (11.91) 149 33.9 (8.48) 
Intervention group 37 31.1 (12.53) 34 33.9 (8.69) 
Control (Total) 123 31.7 (11.71) 115 33.9 (8.46) 
Pupil level control group 55 33.3 (12.56) 50 36.6 (7.04) 
School level control group 68 30.4 (10.91) 65 31.8 (8.90) 


On average, for both cohorts 1 and 2, KS4 / GCSE attainment is observed to be higher within the two 
intervention schools8 compared with the two control schools. Whilst the pattern is consistent, it is 
stronger within cohort 1 compared with cohort 2. For both cohorts 1 and 2, within the two intervention 
schools, mean attainment is slightly higher for the pupil-level control compared with the intervention 
group. 


There were a number of issues with the measurement of the primary outcome. What is evident in 
Table 2.4.1 is that the cohort 1 pupil-level control group had greater variation (as measured by the 
standard deviation) in the original KS4/GCSE attainment variables compared with the intervention and 
school level control groups. Closer inspection of the KS4/GCSE outcome identified two members of 
the pupil-level control group that are rather ‘distinct’ from other pupils. These two cases are recorded 
as taking one KS4/GCSE examination but attaining no GCSE points and are notable outliers in the 
KS4/GCSE attainment distribution9. Table 2.4.1 shows that when these two outlying cases are 
excluded from the analyses, for the pupil level control group, the mean attainment increases and the 
variation reduces - which reflects the (negative) skewing effect that the two outlying cases had on 
estimating mean KS4/GCSE attainment. 


8 This is the combined intervention and pupil-level control groups (i.e. all pupils within the two intervention 
schools). 


9 The two cases are 4 standard deviations below the cohort 1 KS4/GCSE mean and two standard deviations 
below the next case within the KS4 / GCSE distribution. 
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For cohort 2, the variation in KS4/GCSE attainment is seen to be similarly large across all groups. 
Closer inspection revealed 11 instances where a participant was recorded with a mean KS4/GCSE 
score of zero. Nine of these related to participants who are recorded as not sitting any KS4/GCSE 
assessments and two where the participant took one KS4/GCSE assessment but did not attain a 
grade. These cases were found across the treatment groups with three in the intervention group, five 
in the pupil level control group and three in the school level control group. These 11 cases are also 
identified as low outliers from the bulk of the KS4/GCSE distribution for cohort 2 but are less ‘distinct’ 
than the two low outlying cases found in cohort 17°. Table 2.4.1 shows that when these 11 outlying 
cases are excluded from the analyses, the mean attainment increases and variation reduces across 
all of the treatment groups. 


The two low outliers in cohort 1 and 11 low outliers in cohort 2 result in a creating a negative skew 
within the KS4 attainment distributions. To assess the effect of this negative skew has on estimating 
the impact of ThinkForward on KS4/GCSE attainment within this pilot, analyses were conducted of the 
variables both including and excluding the outlying cases. 


It is important to note that the patterns shown in Tables 2.4.1 do not take account of the clustering of 
pupils within schools and make no attempt to correct for the baseline imbalance discussed in section 
2.3. For these reasons, it is not valid to draw conclusions about the impact of ThinkForward on KS4 
attainment. The models specified in sections 2.4.6 and presented in section 3.3 do take account of 
clustering and also include explanatory variables to try to correct for the baseline imbalance and these 
represent a more valid and reliable estimate of impact for the ThinkForward programme on 
KS4/GCSE attainment. 


2.4.2 Secondary Outcome - Unauthorised Absences (NPD) 


Unauthorised absence data was obtained from the NPD for both cohorts. For cohort 1, the outcome 
variables measure unauthorised absences during the spring and summer terms of the 2013/14 
academic year. For cohort 2, the variables measure unauthorised absences during the spring and 
summer terms of the 2013/14 and 2014/15 academic years. 


Spring and summer terms were selected so that the time period fitted with the delayed start of the pilot 
trial. At the end of 2013/14, participating pupils will only have had around six months of the 
programme and so the outcomes at this stage capture short term impact in terms of unauthorised 
absences. At the end of 2014/15, participating pupils will have had around 18 months of the 
programme and so the outcomes here capture longer term impact. 


Unauthorised absences in spring/summer 2012/13 were used as a pre-baseline variable within the 
regression models used to evaluate the impact of ThinkForward. 


Appendix 5 summarises how the unauthorised absence outcome variables were derived using NPD 
data. Unauthorised absence was measured as a (percentage) scale and a simplified binary 
categorical variable. First, the percentage of half sessions missed due to unauthorised absences in 
2012/13, 2013/14 and 2014/15 was derived for each participant. These percentage outcomes 
exhibited a strong positive skew which may result in biased estimates within the regression analyses. 
After exploring some possible data transformations that might reduce the skew, it was decided to 
remove it completely by creating a second simpler unauthorised absence outcome variable. 
Specifically, a binary version of the outcome that identified whether a participant was recorded to have 
missed any (1+) half sessions due to unauthorised absence (=1) or not (=0) was created for absences 
in 2012/13 (pre baseline), 2013/14 (Short term outcome) and 2014/15 (longer term outcome). 


10 The 11 cases are 2.7 standard deviations below the cohort 2 KS4/GCSE mean and one standard deviation 
below the next case within the KS4/GCSE distribution. 
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Table 2.4.2 statistically summarises the two unauthorised absence outcome variables for cohorts 1 
and 2 across the treatment groups and three academic years. 


Table 2.4.2: Unauthorised Absences in spring & summer terms*: 
2012/13 (prior to baseline / randomisation), 2013/14 & 2014/15 


Original scale variable Transformed binary variable 
% Half sessions missed % with 1+ session missed 
(Oxe) ato) gam & 
12/13 13/14 14/15 12/13 13/14 14/15 
Y10 Y11 nla As) Y11 
Intervention 40 4.3% 3.5% / 70% 48% / 
Pupil-level control 75 5.2% 3.1% / 73% A9% / 
eenoollevel 66 2.7% 3.7% / 53% 42% / 
control 
Total sample 181 4.1% 3.4% / 65% 46% / 
fo} ae flare exer Vom lat lel(s Transformed binary variable 
% Half sessions missed % with 1+ session missed 
[oxe) ato) amr 
12/13 13/14 14/15 12/13 13/14 14/15 
Yy9 Yy10 Yi1 Yy9 A Ko) Y1i1 
Intervention 35 4.0% 4.0% 5.8% 89% 77% 51% 
Pupil-level control 54 4.1% 3.8% 3.3% 72% 56% 37% 
School-level 65 3.0% 3.4% 3.0% 60% 68% 46% 
control 
Total sample 154 3.6% 4.7% 3.7% 71% 66% 52% 


* A listwise deletion of missing values is used for Table 2.4.2. For cohort 1, a complete sample of 181 was provided for cohort 1 
in both 2012/13 and 2013/14. However, for cohort 2 complete detail on unauthorised absences for all 162 of the sample was 
provided for 2012/13 and 2013/14 but for 2014/15, details for only 154 participants were provided. Therefore, for cohort 2, the 
sample is restricted to just the 154 pupils with complete details for all three years. Appendix 5 provides further detail including 
statistical summaries of the original (raw) variables. 


A pattern of declining unauthorised absences is evident for both cohorts and across most of the 
treatment groups: the proportion of sessions missed as unauthorised and instances of unauthorised 
absences are observed to decline between Y9 and Y11. For cohort 2, the intervention group are seen 
to have the highest likelihood and incidence of unauthorised absences compared with control groups 
but patterns are more similar in cohort 1. 


As with KS4 attainment it is important to note that the patterns shown in Tables 2.4.2 do not take 
account of the clustering of pupils within schools and make no attempt to correct for the baseline 
imbalance discussed in section 2.3. For these reasons, it is not valid to draw conclusions about the 
impact of ThinkForward on unauthorised absences using Table 2.4.2. The models specified in 
sections 2.4.6 and presented in section 3.3 take account of clustering and include explanatory 
variables to try to correct for the baseline imbalance and represent a more valid and reliable estimate 
of impact for the ThinkForward programme on unauthorised absences. 
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2.4.3 Secondary Outcomes collected via surveys 


Data on the remaining two secondary outcomes (i.e. ThinkForward mind sets and future educational 
expectations) were collected through the participant longitudinal survey; conducted at baseline in 
November 2013, survey 2 in June 2014 and survey 3 in May 2015. As shown below, response to the 
second survey was very low, particularly for cohort 1. This can be explained to some extent by the late 
timing of this survey (June 2014) when many Y11 pupils were no longer regularly attending school. 
This led us to adapt our approach for the final survey to move it earlier in the academic year (May 
2015). Whilst this does seem to have resulted in improved response rates, the problem of non- 
response remained sizable. 


The survey instrument chosen was a self-completion paper questionnaire". A self-completion paper 
questionnaire was chosen over an online version as it was judged to be less risky in terms of non- 
response because it does not rely on pupils’ access to the Internet. 


Key methodological issues which would be important to address in any similar future trials are noted 
here, and then reflected on again in section 5. ThinkForward staff and coaches acted as gatekeepers 
or a ‘hub’ between us and the schools and were responsible for overseeing the data collection for all 
four schools. ThinkForward were responsible for recruiting the four London schools and thus 
developed a strong relationship with these schools, all of which were located in ThinkForward’s local 
geographical area. As evaluators, our relationship with the schools was less regular and more distant. 
As a consequence, ThinkForward staff and coaches were directly involved in the data collection, in the 
gathering of completed questionnaires and in getting the completed data to us for data entry and 
analyses. Even the qualitative interviews to teachers and pupils were commonly arranged through 
ThinkForward or ThinkForward coaches. 


The choice of self-completion questionnaires with the data collection process supervised by 
ThinkForward staff and coaches may have been cost-effective. However, this method led to a number 
of (non-completely anticipated) problems, the potential impact of these on data quality is discussed 
below. 


The involvement of ThinkForward staff and coaches in the data collection raises two main concerns. 


The first concern relates to delays in data collection and processing. At baseline it was around one 
month between when the data collection was completed in London and when we received the 
completed questionnaires in Sheffield. For the second survey this time lag was even greater. We 
adapted the data collection approach for the final survey. For the final survey, data collection took 
place a month earlier than the second survey to try and boost response. Additionally, the process 
evaluation interviews with coaches and pupils were arranged so that they took place just following the 
data collection period for the final survey. This was to allow us to collect the completed questionnaires 
close in time to when they had been completed by pupils. However, in reality only some of 
questionnaires were available to be collected and some were reported by a coach to have been taken 
away from the school by a ThinkForward manager and so once again it was over a month before we 
had all of the questionnaires for processing in Sheffield. 


The second concern relates to the quality of the data collected. An example of the coaches’ potential 
impact on data quality came from informal discussions around the process evaluation. In these 
discussions it was pointed out that one coach had made some pupils complete the (final) survey 


questionnaire more than once because they had felt that the pupil had "not taken it seriously"! When 


11 See Appendix 3 for a copy of the baseline questionnaire. 
12 The coach illustrated this by saying that some pupils had simply ‘ticked all the same answers without reading 
the words’. 
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processing the final survey, we noticed a number of duplicate questionnaires (from both intervention 
schools, although more so from one) - where the same pupil had two different questionnaires. This 
suggests that the problem of duplicated questionnaires is not restricted to one coach. The only option 
we had when finding a duplicate questionnaire was to set all responses for this pupil to be missing. It 
still remains possible that some of the data we accepted from the final survey came from pupils who 
re-did the questionnaire but both completions were not included into the completed questionnaire 
batches sent to Sheffield. It also remains a possibility that the practice of getting pupils to complete the 
survey more than once could have occurred at baseline or the second survey. Whilst we have no 
direct evidence of any conscious attempt of trial subversion resulting from the close involvement of 
coaches in the survey data collection, the above examples casts some doubt on the suitability of 
coaches for data collection. We reflect further on this in the conclusions of this report. 


A further sizable problem with data collected through the longitudinal survey was non-response. Table 
2.4.3 shows the sample sizes and response rates for each survey and each treatment group. In the 
two intervention schools response rates are between 64% and 93%. The response rate of the 
intervention group (75%) is higher than that of the within-school control group (64%) in the case of 
cohort 1. However, in the case of cohort 2, the opposite is observed, i.e. that the within-school control 
group has a higher response rate than the treatment group (93% vs 74%). 


Table 2.4.3: Survey Response - by treatment group and cohort 


Baseline Survey Survey 2 Survey 3 
November 2013 June 2014 WE Ward 0 Be) 


n= n (% response) n (% response) 


COHORT 1 (completed Y11 in 2013/14) 


Intervention group 40 30 (75%) n/at 
Pupil level control group 75 48 (64%) n/a 
School level control . 

group 66 1 (2%) n/a 
Total 181 79 (44%) n/a 


COHORT 2 (completed Y11 in 2014/15) 


Intervention group 38 28 (74%) 24 (63%) 
Pupil level control group 55 51 (93%) 37 (67%) 
School level control . 

group 69 21 (30%) 38 (55%) 
Total 162 100 (62%) 99 (61%) 


1 - Cohort 1 completed Y11 in 2013/14 and did not participate in survey 3. 


Table 2.4.3 also shows that in the school-level control group the response rate never goes beyond 
55%. High non-response is of particular concern for the second survey, for cohort 1 who completed 
Y11 in 2013/14, the response rate is virtually zero (only one questionnaire out of 66 was returned) and 
for cohort 2, who were in Y10 during 2013/14, it is very low at 30%. 


The response rates in Table 2.4.3 shows that the adapted approach did seem to result in better 
response for the final survey. However, a sizable non-response problem remains, especially for the 
school-level control group. 
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The data collection and non-response problems mean that considerable caution is advised when 
interpreting findings relating to the secondary outcomes collected via the longitudinal surveys. For this 
reason, the impact analyses for these outcome variables will be purely descriptive. For the Future 
Educational Expectations and ThinkForward mind set outcome variables, the use of statistical 
modelling risks over-stating the reliability and validity of these outcome variables given the data 
collection and non-response problems discussed above. Therefore, analyses of these secondary 
outcome variables will be confined to simple bivariate tables. These tables will not provide the same 
standard of evidence as the regression approach adopted for KS4/GCSE attainment and unauthorised 
absence NPD outcomes but serve to reflect the relatively lower standard of evidence that they 
provide. 


2.4.3.1 Secondary Outcome: Educational expectations 


Following early inception meetings, the future education expectations items were developed by us in 
close collaboration with colleagues at the University of Essex. It comprised four questions, all asking 
the students to indicate, on a scale from 0 to 100, the perceived likelihood of i) continuing in education 
after Y11, ii) applying to university ili) securing a place in a university conditional on applying iv) 
graduating from university. 


The educational expectations variables are presented in Appendix 6. 


2.4.3.2 Secondary Outcome: ThinkForward ‘Aspiration’ and 'Determination' mind sets 


The questionnaires also included a series of questions designed by ThinkForward to measure 
perception and attitudes. ThinkForward aggregated the answers to these questions into 14 variables, 


which were meant to measure eight different “mind sets” and employability constructs+?, 


Prior to randomisation, ThinkForward selected two of the eight mind sets (the ‘aspiration’ and 
‘determination’ mind sets) for inclusion as secondary outcomes in the impact analysis. Each of the 
ThinkForward mind set variables was constructed by aggregating the answers to five different Likert 
scale questions. This was done by assigning to each question a score equal to the point in the five- 
item Likert scale, and then summing up these scores over the five relevant questions. The resulting 
variables are thus measured on a new scale ranging from 5 (when the student has given the value of 
one to all five five-items Likert scale questions) to 25 (when the student has given the value of five to 


all five five-items Likert scale questions)“. 


Mode detail on the derivation of the ThinkForward mind set variables and the statistical summaries are 
presented in Appendix 7. 


Additional caution is advised for these ThinkForward mind set outcome variables. The outcomes were 
designed by ThinkForward as part of the coaching programme and so are more ‘inherent to treatment’ 
(Slavin & Madden, 2008) than the other three outcome areas. Since this trial commenced, the 
Educational Endowment Foundation have adopted a stronger line on the use of measures that might 
be deemed ‘inherent to treatment’ (EEF, 2014). The position now is that "Measures should always be 
pre-specified and not inherent to treatment" (EEF, ibid p13). In the case of the ThinkForward mind set 
outcomes, these were pre-specified before randomisation took place but clearly are open to bias from 
being inherent to the ThinkForward programme. 


13 These included self-belief, positive thinking Flexibility, Appetite for Learning, Understanding Emotion, Managing 
Emotion, Effective Communication, People Skills, Teamwork, Finding Solutions, Building a Positive Network and 
Planning & Organising - see ThinkForward Training Manual. 

14 See Appendix 7 for more detail on these and the construction of the ThinkForward aspiration and determination 
mind set secondary outcome measures. 
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2.4.4 Analysis Plan 


The protocol for this pilot specified the use of a difference in difference (DD) regression modelling 
approach. A DD approach is suited only to repeated measures outcome variables. The analyses 
undertaken diverged from what was specified in the protocol for the primary outcome (GCSE 
attainment) but a DD approach was conducted for the unauthorised absence outcome variables. 


Due to issues of Survey non-response and data collection problems (see section 2.4.3), for outcome 
variables that drew on data collected using the surveys, the presented analyses are purely descriptive 
and limited to the simple bivariate descriptive tables. 


The National Pupil Database (NPD) provided data on the primary outcome (KS4 / GCSE attainment) 
and one of the secondary outcomes (unauthorised absences) and so these outcome variables did not 
suffer the same non-response and data collection problems. For these outcome variables in addition 
to descriptive analyses, a more comprehensive analytical approach was adopted. This approach 
involved the construction of linear regression models that acknowledged the clustering of participating 
pupils within the four London schools and the inclusion of explanatory variables to try to correct for the 
baseline imbalance discussed in section 2.3. Linear regression was used to analyse the primary 
outcome (GCSE attainment) whilst DD linear regression was used to analyse the unauthorised 
absence secondary outcome. 


Difference in difference (DD) regression modelling includes time as an additional variable and focuses 
on estimating change over time in an outcome variable (and whether this is different for the 
intervention compared with the control groups). DD modelling has two key advantages compared with 
standard regression. First, estimates from DD models are less influenced by baseline differences 
between groups. Second, DD model estimates are based on a greater number of data points than 
standard regression estimates (and so are more precise). These two key advantages of DD need to 
be considered alongside a key assumption of DD; that any observed baseline differences do not have 
an impact on ‘change over time’ for an outcome (in other words, they are assumed to be ‘time 
invariant’). 


For the KS4/GCSE attainment outcome variables, (Standard) regression models were constructed in 
three stages: 


1. Including a (binary dummy) variable that identified whether a pupil was in the ThinkForward 
intervention group (=1) or not (=0) plus a series of binary dummy variables that identified the 
London schools involved in the pilot. 

2. As 1. but also including the (baseline) predicted NEET risk score 

3. As 2 but also including KS3 attainment, gender, FSM, SEN 


The first stage is descriptive. Whilst school-level clustering is acknowledged, at this stage no attempt 
is made to control for the baseline imbalance discussed in section 2.3. The second and third stages 
include the predicted NEET risk score (stage 2) and other factors (stage 3) to try to limit any potential 
bias due to the observed baseline imbalance. These models will show the difference between the 
intervention and control groups once baseline differences* are statistically taken into account. 


Section 3.3.1 presents the findings from the standard regression analyses for KS4/GCSE attainment. 
The estimated impact is taken from the coefficient for the binary dummy variable that identified 
intervention group pupils. This coefficient is converted into (Hedges g) effect size statistic . Details on 


15 Specifically, baseline differences in terms of predicted NEET risk score, KS3 attainment, gender, FSM and SEN 
will be taken into account (or controlled for) within the second and third model stages. 
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Hedges g and how model coefficients were converted can be found in Appendix 8) and full details on 
the standard linear regression models can be found within the Technical Appendix. 


Difference in difference regression models were constructed for the unauthorised absences outcome 
variables: these entered explanatory variables in two stages: 


1. Including an ‘intervention’ variable that identified whether a pupil was in the ThinkForward 
intervention group (=1) or not (=0); a ‘time’ variable that identified whether the time point was 
pre-intervention (=O) or post intervention (=1); the interaction between the intervention and 
time dummy variables (intervention*time) plus a series of binary dummy variables that 
identified the London schools involved in the pilot. 

2. As 2 but also including predicted NEET risk score, KS3 attainment, gender, FSM, SEN 


For the unauthorised absence outcome variables, a difference in difference linear regression approach 
was adopted for the percentage scale version of the outcome and a difference in difference binary 
logistic regression approach was adopted for the simplified binary version of the outcome. 


Section 3.3.2 presents the findings from the standard regression analyses for unauthorised absence. 
The estimated impact is taken from the coefficient for the binary interaction dummy variable. For the 
linear regression analyses of the percentage scale version of the outcome, the coefficient is converted 
into a Hedges g effect size statistic. Details on Hedges g and converting a linear regression model 
coefficient can be found in Appendix 8. For the logistic regression analyses of the simplified binary 
version of the outcome, the coefficient is converted into an odds-ratio effect size statistic. Details on 
odds-ratios converting a logistic regression model coefficient can be found in Appendix 9. The tables 
presented in section 3.3.2 summarise the estimated statistical impact of ThinkForward on 
unauthorised absences but full details on the difference in difference linear and logistic regression 
models can be found within the Technical Appendix. 


The standard and DD regression models specified above were constructed using (NPD) data from 
cohorts 1 and 2. Cohort 1 completed Y11 in 2013/14 when the cohort 1 intervention group pupils will 
have experienced up to six months of the ThinkForward programme. Cohort 2 completed Y11 in 
2014/15 when the cohort 2 intervention pupils will have experienced up to 18 months of the 
ThinkForward programme. 


The GCSE attainment standard regression models relate to GCSE attainment in 2013/14 (for cohort 1) 
and in 2014/15 (for cohort 2). The unauthorised absence DD regression models relate to unauthorised 
absences in spring and summer terms during 2013/14 (for cohorts 1 and 2) and unauthorised 
absences in spring and summer terms during 2014/15 (for cohort 2 only). 


For both cohorts, the standard and DD regression models were constructed using two different 
participant samples. First, the models were constructed using data from the complete sample of four 
London schools and all participating pupils. In these models, the intervention group are compared with 
a control group that is made up of pupils within the two intervention schools and pupils within the two 
control schools. Second, the models were constructed using data from just the two intervention 
schools. In these models, the intervention group are compared with a control group that is made up of 
pupils within the two intervention schools only. Comparing the coefficients from the models based on 
the two different samples will provide some insight the stability of the estimates of impact of the 
ThinkForward programme within a clustered and un-clustered RCT design. 


2.4.7 Capturing ‘spill over’ 


The pilot trial evaluation set out to examine evidence of a spillover effect from the intervention to the 
pupil-level control group. Spill over is defined as a treatment (or intervention) effect that might ripple 
out beyond the specific group it is targeted at. Whilst this might have practical benefits in terms of a 
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positive educational effect beyond pupils directly involved in a programme, it also brings 
methodological problems. If a positive educational effect did spill over from an intervention to a control 
group, the observed difference between the two groups would be smaller. This might lead to 
concluding that the intervention had no measurable impact when in fact the (positive) impact was 
wider than intended. and resulted in the data being contaminated due to spill over. Whilst spill over is 
not completely impossible within cluster-randomised trials where randomisation is at the school level, 
it is more likely to occur when randomisation takes place within a school, at a class or individual pupil 
level. 


Prior to randomisation, participating pupils were asked whether they had any family member and/or 
friends in Y10 or Y11 of the school that they were attending. They were asked to provide their name, 
gender and the year group for each named friend / family member. There was space on the 
questionnaire for up to five family members and five friends. 


Table 2.4.7 summarises the friends and family member links for the two intervention schools for 
cohorts 1 and 2. In both cohorts, 23 pupil-level control group members were identified as having a 
friend or family member in the intervention group in both schools. Whilst the raw numbers are 
identical, links between the intervention and control was observed to be proportionately more likely in 
cohort 2 (42% of the pupil level control group) compared with cohort 1 (31%). 


Table 2.4.7: Friendship & Family links 
Number of pupil-level control group members who were identified in at baseline as having a 
friend / family member within the intervention group. 


Intervention School 1 13 15 
Intervention School 2 10 8 
Total 23 23 
Total size of pupil-level control group 75 55 
% of control group identified as having a friend 31% A2% 
or family member in the intervention group 


To explore the potential statistical impact of spill over, a friends/family links dummy variable was 
included into the regression models. The coefficient for this dummy variable estimates whether control 
group pupils with friends/family in the intervention group had higher (or lower) GCSE attainment / 
unauthorised absences compared with control group pupils without friends/family in the intervention 
group. Additionally, any change in the estimated coefficient of 'impact' observed after the friends/family 
links dummy variable is included into the model provides an indication of contamination due to this 
potential spill over. 


The analyses into direct spill over assume that spill over is transmitted through friends and family and 
ignores other possible sources. For example, from the process evaluation the ThinkForward coaches 
noted that they had been called upon to teach classes within the schools that they worked in (see 
section 4). Coaches stated that they only agreed to teach when one of their Y10 or Y11 intervention 
group pupils was present within these classes. The classes where not exclusive to intervention group 
members, they included other pupils - some of whom may have been within the pupil-level control 
group. This is an example where a spillover effect might be transmitted by the coach. Other examples 
include the possibility of interactions between intervention and pupil-level control group pupils in 
classroom discussions, within the wider school or even external to school (e.g. a sister of an 


Education Endowment Foundation 27 


ThinkForward 


intervention group pupil with a close friend in the pupil-level control group). Essentially, the potential 
sources of spill over are likely to be wider than the direct friends / family links. 


2.5 Process evaluation methodology 


A process and implementation evaluation, completed alongside the piloting of the two RCT designs 
(for within-schools and between school trials) collected qualitative data from ThinkForward coaches, 
the young people involved and school leads. 


The aims of the process evaluation were primarily to assess the implementation of ThinkForward in 
the two intervention schools, and to evaluate fidelity. This was intended to provide a deeper 
understanding of evidence of promise and evidence on feasibility and readiness for trial. Areas of 
focus were: how coaches had been working in the schools from the school lead and coach's 
perspective, how far coaches were able to implement the ThinkForward programme with their cohorts 
of pupils (including issues of access to pupils), how successful coaches and school leads felt the 
programme to be in improving outcomes, and the pupils’ perspective on the programme and the 
coaches. In addition the process evaluation was of particular importance to this pilot because it 
explored the practicalities of pupil level randomisation (e.g. how pupils and parents respond to finding 
out about being placed within the control group) and collected details on pupil peer friendship groups. 


The process evaluation largely focused on the interpersonal relationships between young people and 
coaches, the balance of one to one and group working with the intervention cohort, and the 
relationships between coaches and school leadership teams and environmental factors that impacted 
those relationships. The evaluation team's evidence (see Section 4) is partly focused on the extent to 
which coaches employed the ThinkForward tools themselves, but also focusses on whether the 
coaching interaction was successfully creating the kind of stable relationships desired by the 
intervention. 


Data was collected by the evaluation team at Sheffield Hallam University. The project manager and 
qualitative project director conducted all the process evaluation fieldwork. Table 2.5.1 outlines the data 
collection methods used and the timeline. 


Table 2.5.1: Process evaluation timeline and methods 


Time point Process evaluation activity 


First case study visit to two intervention schools. Interviews with school leads 


April 2014 and coaches 


Second case study visit to two intervention schools. Focus groups with Y11 
pupils in cohort 1 in both intervention and control groups. 


June 2014 


January 2015 Telephone interviews with coaches in the two intervention schools. 


January 2015 Telephone interview with school lead in one control school. 


Third case study visit to two intervention schools. Interviews with coaches and 
May 2015 school lead. Focus groups conducted with Y11 pupils in cohort 2 in both 
intervention and control groups. 


January 2016 Telephone interview with ThinkForward representative. 


The majority of data for the process evaluation was collected during visits to the two intervention 
schools at three time points between April 2014 and April 2015. In School B it was not possible to 
interview the school lead during the second visit due to the school lead being promoted and becoming 
increasingly busy. 


Education Endowment Foundation 28 


ThinkForward 


In addition to this, shorter ‘catch up’ telephone interviews were conducted with the coaches in January 
2015. Also in January 2015, a telephone interview was conducted with a school lead at one of the 
control schools. The other control school was contacted and an interview was arranged, however at 
the start of the interview the person identified as the school lead explained that he was not aware of 
ThinkForward and was therefore not able to be interviewed about this. 


Although a number of interviews were conducted by telephone (‘catch-up’ coach interviews and 
interview with control school lead) most were conducted in person during fieldwork visits. Face to face 
interviews were deemed to be the most appropriate methods of data collection in order to; firstly visit 
the school to see the context in which coaches were working, and, in addition, to be able to ask in- 
depth questions following a semi-structured interview schedule which a face to face interview 
facilitates. Conducting focus groups with pupils was useful, particularly for the age group involved, in 
order to build a rapport and allow conversations and ideas to emerge through pupils sharing thoughts 
with each other as well as the focus group facilitator. 


Interviews were fully transcribed and then analysed thematically. This analysis was both deductive 
and inductive, using a combination of our interview questions (used in the interview schedules and 
themes emerging from the interview data itself. This created a series of additional codes and themes 
which formed the basis for the structure of reporting the process and implementation findings. 


2.6 Pilot of trial methodology 


The pilot adopted a design that randomised at both the school and pupil levels to create a mixed 
research design. As detailed in section 2.4, the pilot applied standard regression and difference in 
difference regression modelling within analyses exploring evidence of promise for the ThinkForward 
programme. In addition to this, detail on friendship and family links between the intervention group and 
pupil-level control group was used to explore evidence of ‘spill over’ (See section 2.4.9). 
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Date Activity 


April - Oct 2013 School Recruitment, Protocol Development 
Schools sign MoU, seek opt-out consent for survey and NPD linkage, 
Oct & Nov 2013 Baseline Survey Data Collection (future educational expectations, 
ThinkForward mind sets & friends / family links) 
Nov 2013 School Level Randomisation 
Pupil level randomisation, public registration of pilot trial, first NPD 
Dec 2013 request (KS3 attainment, 2012/13 unauthorised absences & pupil 
background) 
: Initial Process Data Collection (interviews with school leads and 
April 2014 coaches) 
Second process Data Collection (pupil focus groups) 
June 2014 Second Survey data collection (future educational expectations & 
ThinkForward mind sets) 
November 2014 Second NPD request (2013/14 KS4/GCSE attainment for cohort 1; 
unauthorised absences in 2013/14 for cohorts 1 and 2) 
Third process Data Collection (telephone interviews with coaches & 
aonikaly 220i control school leads) 
Fourth Process Data Collection (interviews with coaches & school leads 
f ith il 
May 2015 and ocus groups with pupi s) . . 
Final Survey data collection (future educational expectations & 
ThinkForward mind sets) 
November 2015 Final NPD request (2013/14 KS4/GCSE attainment for cohort 1; 
unauthorised absences in 2013/14 for cohorts 1 and 2) 
Dec 2015 Final NPD request (2013/14 KS4/GCSE attainment for cohort 1; 
unauthorised absences in 2013/14 for cohorts 1 and 2) 
Final process data collection § (interview with ThinkForward 
moments representative) 
2.8 Costs 


The cost information was provided by the developer of ThinkForward. In the costs section, we provide 
the costs associated with implementing the intervention and the costs associated with the evaluation. 
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3. Evidence of promise 


Within this pilot, evidence of promise was examined using nine outcome variables across four 
outcome areas, within two academic years and amongst two cohorts of pupils. 


The pupils in cohort 1 were in Y11 at the start of the trial in January 2014 and experienced up to six 
months of ThinkForward within the trial period up to their KS4/GCSE assessments in summer 2014. 
The pupils in cohort 2 were in Y10 at the start of the trial and experienced up to 18 months of 
ThinkForward within the trial period up to their KS4/GCSE assessments in summer 2014. 


The outcome variables were obtained from the NPD and a (three wave) longitudinal pupil survey: 


e Primary Outcome - KS4/ GCSE attainment measured using the mean KS4/GCSE points per 
KS4 assessment in 2013/14 (cohort 1) and 2014/15 (cohort 2). 
e Unauthorised absences during the academic year prior to the trial and the first and second 
year of the trial. Unauthorised absences were measured in two ways: 
o %of half sessions missed as unauthorised during the spring & Summer terms in 
2013/14 (cohorts 1 and 2) and 2014/15 (cohort 2 only). 
o whether pupils had missed one or more half sessions recorded as unauthorised 
during the spring & summer terms in 2013/14 (cohorts 1 and 2) and 2014/15 (cohort 2 
only). 


Longitudinal Pupil Survey: 


e Future Educational Expectations of participants at baseline, in June 2014 (for cohorts 1 and 2) 
and in May 2015 (for cohort 2 only). Specifically, expectations relating to staying in full time 
education after Y11, applying to university, getting a place at university and graduating. 

e ThinkForward ‘mind sets' from participants at baseline, in June 2014 (for cohorts 1 and 2) and 
in May 2015 (for cohort 2 only). Specifically, two 'mind sets' were included amongst the 
variables; Aspiration and Determination. 


Due to the severe non-response and notable data collection issues with the survey discussed in 
section 2.4.3, analyses of the outcome variables that drew on data through the longitudinal surveys 
are purely descriptive. This seriously undermines the validity of drawing conclusions on the impact of 
ThinkForward relating to these outcomes (Future Educational Expectations and ThinkForward ‘mind 
sets’). As discussed in section 2.4.6, the analyses of outcome variables extracted from the NPD are 
more comprehensive. For these outcome variables (GCSE attainment and unauthorised absences), 
standard and difference in difference regression models that took account of the clustering of pupils 
within the four London schools were constructed. This means that drawing conclusions of impact 
relating to these two outcome areas will be more valid but caution is still advised given the small scale 
of this pilot. 


3.1 Participants 


As detailed in section 2.4, within the two intervention schools, a stratified randomisation strategy was 
applied to select the pupils who would receive ThinkForward. Those not selected formed the pupil- 
level control group. Following randomisation two intervention group pupils in cohort 2 were dropped 
from the evaluation. This is because they were both permanently excluded from school. 


3.2 Randomisation 


The school and pupil level randomisation are detailed in section 2.3. As shown and discussed in 
section 2.3, randomisation resulted in a baseline imbalance. On average, the intervention group had a 
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higher predicted NEET risk score and lower KS3 attainment compared with the control group(s). The 
intervention group were also less likely to be 'FSM', less likely to be female and more likely to have a 
SEN with a statement or School Action Plus when compared with the pupil-level control group. In this 
pilot, we addressed the observed baseline imbalance through the use of baseline covariates and 
difference in difference modelling (See Section 2.4.6). 


3.3 Outcomes and analysis 


This section is organised into sub-sections that focus on each of the four outcome areas: KS4/GCSE 
attainment, unauthorised absences; future educational expectations and ThinkForward mind sets. 


3.3.1: Primary Outcome: KS4/ GCSE Attainment 


Table 2.4.1 in section 2.4.1 shows KS4/GCSE attainment*® across the three pupil treatment groups. 
On average, for both cohorts, KS4/GCSE attainment is seen to be higher within the two intervention 
schools compared with the two control schools. The pattern is consistent for both cohorts but is 
stronger for cohort 1. Within the two intervention schools, on average for both cohorts, KS4/GCSE 
attainment is seen to be higher within the pupil-level control compared with the intervention group - 
and the size of this difference is consistent for both cohorts 1 and 2. 


For the impact analyses, standard linear regression models were constructed that included covariates 
to try to address the baseline imbalance, adjusted the standard error estimates for the clustering of 
pupils into schools and included school fixed effects terms; as specified in section 2.4.6. 


Table 3.3.1a summarises the estimated Hedges g effect size statistics for the GCSE attainment 
models for cohorts 1 and 2. Specifically, Table 3.3.1 presents the estimated intervention coefficient 
standardised into (Hedges g*’) effect size statistics with 95% confidence intervals’®. Estimated 
Hedges g effect sizes are shown for the models at each stage of construction (for more details see 
section 2.4.6). Model estimates obtained from analyses of data from the complete sample of four 
London schools and estimates obtained from analyses of data from just the two intervention school 
subsample (see section 2.4.6). Finally, because of a number of low outlying cases within the KS4 
attainment variable (see section 2.4.1), Table 3.3.1a shows estimated effect sizes for samples that 
include and exclude these cases. Full details on these models are available in the Technical 
Appendix. 


16 KS4/GCSE attainment is measured using the mean KS4/GCSE score per exam. 

17 See Appendix 8 for detail on Hedges g effect size statistics and how the model coefficients are converted into 
them from Cohen's d effect size statistics. 

18 Extracted from output generated using STATA regression with robust standard errors, clustered at the school 
level and including dummy variables for each of the schools. 
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Table 3.3.1a: Summary of KS4/GCSE Attainment models. 

Model intervention group coefficients converted into (Hedges g) effect sizes 

For all models, the standard error has been corrected for school-level clustering and school- 
level dummy variables are included. 


Original / Raw mean KS4 Excluding low outliers (with zero KS4 
score per exam score per exam) 


Tal ccyavc=yayicey a] 
(oxol-yai(er(=)almexe)anV(cdarzve| 
into Hedges g effect 

sizes (with 95% Cl) 


Cohort 1 (KS4 in 2013/14) 


Intervention coefficient 


p<0.05 converted into Hedges g p<0.05 
effect sizes (with 95% Cl) 


Complete sample (all 4 schools) 


Stage 1* -0.17 (-0.58; +0.23) No -0.31 (-0.13; -0.48) Yes 
Stage 2* +0.15 (-0.77; +1.07) No +0.02 (-0.61; +0.66) No 
Stage 3* +0.07 (-0.14; +0.27) No +0.02 (-0.10; +0.13) No 


Just 2 intervention schools 


Stage 1* -0.16 (-2.03; +1.70) No -0.31 (-1.18; +0.56) No 
Stage 2* +0.17 (-4.61; +4.96) No +0.04 (-3.51; +3.59) No 
Stage 3* +0.02 (-0.81; +0.86) No -0.02 (-0.54; +0.50) No 


Cohort 2 (KS4 in 2014/15) 


Complete sample (all 4 schools) 


Stage 1* +0.13 (+0.07; +0.18) Yes +0.20 (-0.78; +1.18) No 
Stage 2* -0.05 (-0.35; +0.24) No -0.03 (-0.57; +0.52) No 
Stage 3* -0.03 (-0.40; +0.33) No -0.05 (-0.46; +0.36) No 


Just 2 intervention schools 


Stage 1* +0.12 (-0.13; +0.37) No +0.22 (-5.03; +5.47) No 
Stage 2* -0.08 (-1.48; +1.31) No -0.07 (-2.68; +2.55) No 
Stage 3* -0.04 (-1.91; +1.83) No -0.04 (-1.98; +1.89) No 


*Please see section 2.4.6 for more detail on these model stages and the Technical Appendix for full 
model details. To summarise here: 


e Atstage 1 the models include a dummy variable identifying whether a pupil was in the 
intervention group. 

e Atstage 2 the models include the baseline predicted NEET risk score to try to statistically 
correct for the baseline imbalance discussed in section 2.3. 


At stage 3 the models also include further baseline covariates to try to further correct for the baseline 
imbalance - KS3 attainment, gender, FSM & SEN. (i.e. KS3 attainment refers to KS Level 3 English, 
Maths and Science teacher assessments) see Table 2.3.1. 


For both cohorts, the estimated impact of participation in ThinkForward on GCSE attainment is very 
close to zero and not statistically significant. This was found to be the case with the original 
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KS4/GCSE attainment variable and when the low outlying KS4/GCSE attainment cases were 
excluded from the analyses. 


We therefore conclude that we found no evidence of the impact of ThinkForward relating to 
improvements in KS4/GCSE attainment in the short (cohort 1, 6 month) and longer (cohort 2, 18 
months) term. 


The impact analyses presented in Table 3.3.1a ignores the potential issue of spill over. Table 3.3.1b 
provides a statistical look at the potential existence of spill over transmitted through friendship and 
family links between the control and intervention group (see section 2.4.7). Table 3.3.1b replicates the 
(stage 3, see section 2.4.6) estimated Hedges g intervention effect size statistics shown in Table 
3.3.1a. Below this, Table 3.3.1b shows the estimated intervention effect size once the friends/family 
links dummy variable was included into the model. 


For cohort 2, no statistical evidence of spill over (through friends/family links) was observed. The 
friends/family links dummy variable was not statistically significant across all models and the inclusion 
of the friends/family dummy variable does not result in a notable change in the intervention effect size. 
The impact of including the friends/family links dummy variable is observed to be stronger within the 
models that ignore the low outlying GCSE attainment cases but none reached statistical significance. 


However, for cohort 1, some evidence of potential spill over and the potential contaminating effect of 
spill over was observed. The coefficient for the friends/family dummy variable was positive and 
statistically significant for two of the four cohort 1 models shown in Table 3.3.16. Control group pupils 
with known friends/family links with the intervention group are observed to attain higher than control 
group pupils without friends/family links. Further, when this is taken account of within the analyses the 
intervention effect size is consistently positive and statistically significant for models based on the full 
sample of all four schools. 


Some caution is advised in drawing strong conclusions about spill over and GCSE attainment from the 
analyses presented in Table 3.3.1b. Specifically, there are four reasons for caution. First, the 
coefficient for the friends/family dummy variable was positive and statistically significant for models 
that ignored the low outlying GCSE/KS4 attainment cases. When these cases are excluded from the 
analyses, the coefficient remains positive but is smaller and not statistically significant. Second, the 
Statistically significant intervention effect sizes are observed only within the models that include data 
from all four schools. The intervention effect sizes for the models confined to data from just the two 
intervention schools are seen to be positive and to increase when the friends/family dummy variable is 
included but do not reach statistically significance. Third, cohort 1 will have had only up to six months 
of ThinkForward before completing their KS4/GCSE assessments in summer 2014. The lack of 
statistical evidence of spill over observed for cohort 2 (who will have had up to 18 months of 
ThinkForward) serves to question the veracity of evidence for spill over within cohort 1. Finally, this is 
a small scale pilot and as such caution is advised in drawing strong conclusions from the presented 
findings. If the observed patterns were consistent across cohorts, samples and with respect to the low 
outlying KS4/GCSE cases, there would be stronger statistical evidence of spill over. - but this is not 
the case here and so, caution is advised. 


In summary, whilst some statistical evidence of spill over and the potential contaminating effect of spill 
over in estimating the impact of ThinkForward on KS4/GCSE attainment was found, this was confined 
to cohort 1 and should be treated with caution. This does, however, need to be considered alongside 
evidence of potential spill over through ThinkForward coaches that emerged from the process 
evaluation and is discussed in section 4. Future trials using within school (pupil level) randomisation 
may want to collect evidence of spill over more systematically using items on a survey for control 
group members. 


Education Endowment Foundation 34 


ThinkForward 


Table 3.3.1b: KS4/GCSE Attainment models - a look at spill over 

Model intervention group coefficients converted into (Hedges g) effect sizes 

Stage 3 models and including the friends/family links dummy variable. 

For all models, the standard error has been corrected for school-level clustering and school- 
level dummy variables are included. 


Original / Raw mean KS4 score Excluding low outliers (with zero KS4 
per exam score per exam) 


Intervention 
coefficient Taciavceyayicey a 
converted into <0.05 oxel=yii(edi=ialmexe)anyiclancve! 
Hedges g effect paw into Hedges g effect 
sizes (with 95% sizes (with 95% Cl) 
co) 


p<0.05 


Cohort 1 (KS4 in 2013/14) 


Complete sample (all 4 schools) 


Stage 3* +0.07 (-0.14; +0.27) No +0.02 (-0.10; +0.13) No 

+ f/f dummy** | +0.18 (+0.01; +0.35) Yes +0.12 (+0.01; +0.23) Yes 
f/f dummy 

ae +0.34 sds Yes +0.30 sds No 


Just 2 intervention schools 


Stage 3* +0.02 (-0.81; +0.86) No -0.02 (-0.54; +0.50) No 

+ f/f dummy** | +0.12 (-0.58; +0.83) No +0.08 (-0.01; +0.16) No 
f/f dummy 

ae +0.31 sds Yes +0.29 sds No 


Cohort 2 (KS4 in 2014/15) 


Complete sample (all 4 schools) 


Stage 3* -0.03 (-0.40; +0.33) No -0.05 (-0.46; +0.36) No 

+ f/f dummy** | +0.12 (-0.13; +0.36) No -0.04 (-0.37; +0.29) No 
f/f dummy 

ener -0.34 sds No -0.03 sds No 


Just 2 intervention schools 


Stage 3* -0.04 (-1.91; +1.83) No -0.04 (-1.98; +1.89) No 

+ f/f dummy** | +0.09 (-0.83; +1.02) No -0.05 (-1.28; +1.17) No 
f/f dummy 

ener -0.28 sds No +0.02 sds No 


Please see section 2.4.6 for more detail on these model stages, section 2.4.7 for detail on the 
friends/family dummy variable and the Technical Appendix for full model details. To summarise here: 


e Atstage 3: the models include a dummy variable identifying whether a pupil was in the 
intervention group, the baseline predicted NEET risk score, KS3 attainment, gender, FSM & 
SEN to try to statistically correct for the baseline imbalance discussed in section 2.3. 
Converted into a Hedges g effect size statistic 
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e +f/f dummy: the models also include a dummy variable that that identifies whether a control 
group pupil has a friend or family member in the intervention group (See section 2.4.7) This 
dummy variable is added to the stage 3 KS4/GCSE models - the effect of spill over. Converted 
into a Hedges g effect size statistic 

e = f/f dummy coef: this is the coefficient for the friends/family dummy variable converted into a 
Hedges g effect size statistic - the spill over effect.. 


3.3.2 Secondary Outcome: Unauthorised Absences (spring/summer terms) 


Table 2.4.2 in section 2.4.2 summarises unauthorised absences for the different treatment groups for 
both cohorts. The percentage scale version of the measure captures the rate of unauthorised absence 
across the treatment groups but contained a strong positive skew. The simplified binary version of the 
measure captures the incidence of any unauthorised absence, has no positive skew but is a much 
simpler / blunter measure compared with percentage rate. 


For cohort 1, absences in 2012/13 (pre-baseline) and 2013/14 are shown. For cohort 2, absences in 
2012/13 (pre-baseline), 2013/14 and 2014/15 are shown. 


On average, for cohort 1, unauthorised absences are seen to be higher within the two intervention 
schools compared with the two control schools both in 2012/13 (prior to baseline, when cohort 1 were 
in Y10) and in 2013/14. Rates of unauthorised absences were smaller on average in 2013/14 
compared with 2012/13. 


For cohort 2, a slightly different picture is observed where the intervention group are seen to have the 
highest rates of absence at all three time points but the pupil-level control group are seen to have the 
lowest rates. This is most clearly seen with the binary version of the measure. Rates of unauthorised 
absence also declined over time for the intervention and pupil-level control groups but are seen to rise 
between 2012/13 and 2013/14 for the school-level control group but then fall in 2014/15. 


Table 3.3.2a summarises the difference in difference models for the percentage scale and simplified 
binary versions of the unauthorised absences outcome variables (see section 2.4.2). For the 
percentage scale version of the outcome, Table 3.3.2a presents the estimated intervention coefficient 
standardised into (Hedges g?°) effect size statistics with 95% confidence intervals. Estimated Hedges 
g effect sizes are shown for the models at each stage of construction (See section 2.4.6). For the 
simplified binary versions of the outcomes, Table 3.3.2a presents the estimated intervention coefficient 
standardised into odds-ratio”° effect size statistics with 95% confidence intervals. 


The first thing to note is that across all of the models presented in Table 3.3.2a, none of the estimates 
of impact reached statistical significance. Comparing estimates from the percentage scale (but 
positively skewed) outcome with those from the simplified binary outcome, a consistent pattern is 
elusive. Across most models, logistic regression estimates from modelling the simplified binary version 
of the outcome measure suggest that ThinkForward is associated with a reduced likelihood of 
unauthorised absences. However, at the same time, across most models, linear regression estimates 
from modelling the simplified binary version of the outcome measure suggest that ThinkForward is 
associated with an increased percentage rate of unauthorised absence. This might relate to the 
influence of high outliers and positive skew on the linear regression estimates, which suggests that 
greater credence should be given to the binary outcome. But given the lack of statistical significance, 
the lack of consistency across all models and the blunt nature of the binary outcome, the findings are 
inconclusive. 


19 See Appendix 8 for detail on Hedges g effect size statistics and how the linear regression model coefficients 
are converted into them. 

20 See Appendix 9 for detail on odds ratio effect size statistics and how the logistic regression model coefficients 
are converted into them. 
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We therefore conclude that we found no evidence of the impact of ThinkForward relating to reduced 
rates and incidences of unauthorised absence in the short (6 month) and longer (18 month) term 


Table 3.3.2a: Summary of unauthorised absence difference in difference linear and logistic 
regression models. Model intervention*time coefficients converted into effect size statistics 
(Hedges g & Odds-Ratio). 


Scale: % of half sessions missed 
as unauthorised 


Binary Transformation (1+ instance of 
unauthorised absence 


Difference in Difference linear 
regression 


Difference in Difference binary logistic 
regression 


Intervention 
coefficient converted 
into Hedges g effect 

sizes (with 95% Cl) 


Taceyavccdaeceyal 
coefficient converted 
into Odds Ratio effect 

sizes (with 95% Cl) 


p<0.05 


Cohort 1 (absences in 2013/14 — when in Y11) 


Complete sample (all 4 schools) 


Stage 1* -0.02 (-0.52; +0.47) No 0.81 (0.23; 2.93) No 

Stage 2* -0.18 (-0.77; +0.41) No 0.69 (0.08; 6.14) No 
Just 2 intervention schools 

Stage 1* +0.14 (-1.61; +1.89) No 1.10 (0.17; 6.97) No 

Stage 2* -0.03 (-2.25; +2.19) No 1.09 (0.06; 19.56) No 


Cohort 2 (absences in 2013/14 - when in Y10) 


Complete sample (all 4 schools) 


Stage 1* +0.05 (-0.37; +0.48) No 0.54 (0.25; 1.20) No 

Stage 2* +0.05 (-0.38; +0.49) No 0.51 (0.19; 1.33) No 
Just 2 intervention schools 

Stage 1* +0.13 (-2.19; +2.45) No 1.01 (0.94; 1.08) No 

Stage 2* +0.13 (-2.22; +2.48) No 1.15 (0.83; 1.59) No 


Cohort 2 (absences in 2014/15 - when in Y11) 


Complete sample (all 4 schools) 


Stage 1* +0.25 (-0.09; +0.58) No 0.42 (0.16; 1.10) No 

Stage 2* +0.25 (-0.08; +0.57) No 0.41 (0.14; 1.20) No 
Just 2 intervention schools 

Stage 1* +0.29 (-0.59; +1.18) No 0.73 (0.35; 1.50) No 

Stage 2* +0.28 (-0.53; +1.09) No 0.74 (0.42; 1.33) No 
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*Please see section 2.4.6 for more detail on these model stages and the Technical Appendix for full 
model details. To summarise here: 


e Atstage 1 the models include a dummy variable identifying whether a pupil was in the 
intervention group, a time dummy variable and the interaction between these two. The 
coefficient for the interaction is the difference in difference estimate of the impact of 
ThinkForward on unauthorised absences.. 

e Atstage 2 the models include predicted NEET risk score, KS3 attainment, gender, FSM & 
SEN to try to statistically correct for the baseline imbalance - 


The impact analyses presented in Table 3.3.2a ignores the potential issue of spill over. Table 3.3.2b 
provides a statistical look at the potential existence of spill over transmitted through friendship and 
family links between the control and intervention group (see section 2.4.7). Table 3.3.2b replicates the 
stage 2 estimated effect size statistics shown in Table 3.3.2a. Below this, Table 3.3.2b shows the 
estimated intervention effect size once the friends/family links dummy variable was included into the 
model. Below this Table 3.3.2b shows the coefficient for the friends/family links dummy variable21. 


The first thing that is striking from Table 3.3.2b is the stability of the difference in difference estimates. 
The inclusion of the friends/family dummy variable results in very little change in the estimated impact 
of ThinkForward on unauthorised absence rates or incidence. The coefficient for the friends/family 
variable was statistically significant for the spring/summer 2013/14 (6 month, short term) outcome for 
both cohorts. This, however, is not in a consistent direction; in 2013/14, friends/family links are 
associated with increased likelihood of unauthorised absences for cohort 1 but reduced likelihood for 
cohort 2. Overall, evidence of spill over relating to reduced unauthorised absences is slim and the 
impact of spill over on estimating the effect of ThinkForward negligible. 


In summary, little/no statistical evidence of spill over and the potential contaminating effect of spill over 
(through friends / family links) in estimating the impact of ThinkForward on unauthorised absences 
was found. This needs to be considered alongside evidence of potential spill over through other routes 
such as the coaches discussed in section 4. 


21 Please note that the coefficient for the friends/family links variable is shown in the original raw units - 
percentage absences or the log-odds (Logit) of unauthorised absence. 
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Table 3.3.1b: Unauthorised absences models - a look at spill over 


Difference in difference model intervention*time coefficients converted into effect size statistics 
(Hedges g and odds-ratios) 


Stage 2 models and including the friends/family links dummy variable. For all models, the standard 
error has been corrected for school-level clustering and school-level dummy variables are included. 


Original % of half sessions Binary Transformation (1+ instance of 
missed as unauthorised unauthorised absence 


Difference in Difference linear Difference in Difference binary logistic 
regression regression 


Intervention Intervention 
coefficient coefficient 
converted into converted into 
Hedges g effect Odds Ratio effect 
sizes (with 95% Cl) sizes (with 95% Cl) 


Cohort 1 (absences in 2013/14 - when in Y11) 


Complete sample (all 4 schools) 


Stage 2* -0.18 (-0.77; +0.41) No 0.69 (0.08; 6.14) No 
+ fif dummy* -0.18 (-0.77; +0.42) No 0.69 (0.07; 6.63) No 
fif dummy coef* -0.04 sds No 2.30 Yes 


Just 2 intervention schools 


Stage 2* -0.03 (-2.25; +2.19) No 1.09 (0.06; 19.56) No 
+ fif dummy* -0.03 (-2.25; +2.20) No 1.11 (0.05; 23.71) No 
fif dummy coef* -0.06 sds No 2.11 Yes 


Cohort 2 (absences in 2013/14 - when in Y10) 


Complete sample (all 4 schools) 


Stage 2* +0.05 (-0.38; +0.49) No 0.51 (0.19; 1.33) No 
+ fif dummy* +0.05 (-0.38; +0.49) No 0.51 (0.19; 1.33) No 
flf dummy coef* +0.26 sds No 0.56 Yes 


Just 2 intervention schools 


Stage 2* +0.13 (-2.22; +2.48) No 1.15 (0.83; 1.59) No 
+ f/f dummy* +0.13 (-2.23; +2.49) No 1.15 (0.90; 1.48) No 
fif dummy coef* +0.26 sds No 0.48 Yes 


Cohort 2 (absences in 2014/15 - when in Y11) 


Complete sample (all 4 schools) 
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Stage 2* +0.25 (-0.08; +0.57) No 0.41 (0.14; 1.20) No 
+ fif dummy* +0.24 (-0.08; +0.57) No 0.41 (0.14; 1.22) No 
fif dummy coef* +0.32 sds Yes 0.87 No 


Just 2 intervention schools 


Stage 2* +0.28 (-0.53; +1.09) No 0.74 (0.42; 1.33) No 
+ fif dummy* +0.28 (-0.50; +1.06) No 0.74 (0.40; 1.38) No 
fif dummy coef* +0.33 sds No 0.90 No 


*Please see section 2.4.6 for more detail on these model stages, section 2.4.7 for detail on the 
friends/family dummy variable and the Technical Appendix for full model details. To summarise here: 


e Atstage 3: the models included the three difference in difference dummy variables, the 
baseline predicted NEET risk score, KS3 attainment, gender, FSM & SEN to try to statistically 
correct for the baseline imbalance discussed in section 2.3. Converted into either Hedges g or 
odds-ratio effect size statistics. 

e + f/f dummy: the models also include a dummy variable that that identifies whether a control 
group pupil has a friend or family member in the intervention group (See section 2.4.7) This 
dummy variable is added to the stage 3 KS4/GCSE models - the effect of spill over. Converted 
into either Hedges g or odds-ratio effect size statistics. 

e = fff dummy coef: this is the coefficient for the friends/family dummy variable converted into 
either Hedges g or odds-ratio effect size statistics - the spill over effect.. 


3.3.3 Secondary Outcomes: Future Expectations 


As explained in section 2.4.3, analyses of outcomes that drew on data from the longitudinal pupil 
survey are limited to simple bivariate tables. Table 3.3.3 presents the mean expectation score for the 
four future expectation outcomes across treatment groups for both cohorts. See Appendix 6 for more 
detail on the future expectations outcomes. 


From Table 3.3.3, evidence of potential impact relating to Future Educational Expectations is limited to 
expectations about university but, because this pattern is found from simple bivariate tables, drawing 
strong conclusions of the impact of ThinkForward from them is not suitable. A sharp decline in 
expectations of applying to university, getting a place at university and graduating is observed for the 
intervention group between baseline and June 2014 which contrasts with expectations within the 
control groups. The pattern is seen for both cohorts 1 and 2 but, for cohort 2, reported university 
expectations are seen to return to be more similar to the control groups by May 2015. In summary, 
with respect to expectations around university, Table 3.3.3 shows some evidence of a short term (6 
month) decline followed by a longer term (18 month) recovery. 
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Table 3.3.3: Summary of Future Educational Expectations* 


Mean scores across treatment groups (cohorts 1 and 2) 
Cohort 1 Cohort 2 


Base June | May 7 June 
n Base 
2014 2015 2014 


Stay in FT Education after Y11 


Intervention group 22 75.4 79.1 / 12 79.6 71.3 93.2 

Pupil level control 36 83.2 81.6 / 30 75.1 85.4 84.1 

School level control 1 - - / 11 86.8 86.4 89.5 
Total Sample 59 78.8 80.6 / 53 78.5 82.4 87.3 


Apply to University 


Intervention group 22 52.0 39.1 / 12 64.5 53.3 54.7 

Pupil level control 36 65.9 59.3 / 26 73.6 73.4 72.9 

School level control 1 - - / 12 63.8 73.3 72.9 
Total Sample 59 61.3 51.6 50 69.1 68.5 64.0 


Get a place at University 


Intervention group 21 65.7 48.1 / 12 54.9 43.3 50.3 
Pupil level control 36 63.5 65.0 / 25 72.4 68.8 67.0 
School level control 1 - - / 11 67.4 67.7 68.5 
Total Sample 58 64.4 58.9 / 48 66.9 62.2 63.2 
Graduate 

Intervention group 21 60.0 46.4 / 12 62.0 45.5 59.0 
Pupil level control 36 68.6 65.4 / 28 77.5 70.1 69.7 
School level control 1 - - / 12 57.4 70.3 68.7 
Total Sample 58 65.3 58.5 / 52 69.3 64.5 67.0 


* The mean scores shown in Table 3.3.3 are based on data from pupils who responded at all time points (two for cohort 1, three 
for cohort 2). This is known as listwise deletion of missing values. Please see Appendix 6 for more detail including the original 
(raw) versions of the variables. 
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3.3.4 Secondary Outcome: ThinkForward ‘Mind Sets' 


As explained in section 2.4.3, analyses of the ThinkForward mind set outcomes are limited to simple 
bivariate tables. Table 3.3.4 presents the mean expectation score for two mind set outcome variables 
across treatment groups for both cohorts. See s Appendix 7 for more detail on the ThinkForward mind 
set outcomes. 


From Table 3.3.4, evidence of impact relating to the ThinkForward designed ‘aspiration’ and 
‘determination’ mind set outcome variables was slim and is based on simple bivariate tables, so 
drawing strong conclusions of the impact of ThinkForward from them is not suitable. For both cohorts 
1 and 2, little difference is seen between baseline and June 2014. However, for cohort 2, increases in 
the mean ‘aspiration’ and ‘determination’ mind set between June 2014 and May 2015 is observed for 
the intervention group and school-level control group whilst a decline is observed for the within school 
pupil-level control group. Because these reported patterns come from simple bivariate tables, drawing 
strong conclusions of the impact of ThinkForward from them is not suitable. The pattern is stronger for 
the determination compared with the aspiration mind set. 


Table 3.3.4: Secondary Outcome: ThinkForward Mind Sets*: 
Mean Aspiration & Determination scores across treatment groups 


(exe) ato) ame Cohort 2 


Base June May June 
n Base 
2014 2015 2014 


‘Aspiration’ mind set 


Intervention group 22 19.4 18.9 / 12 18.6 18.2 19.3 

Pupil level control 33 18.4 18.8 / 27 18.4 18.9 18.3 

School level control 1 - - / 12 18.4 19.3 19.5 
Total Sample 56 18.8 18.8 / 51 18.4 18.8 18.8 


‘Determination’ mind set 


Intervention group 23 17.4 18.2 / 17 15.4 16.6 17.5 

Pupil level control 32 17.2 17.4 / 27 16.5 16.5 16.3 

School level control 1 - - / 13 17.3 17.8 17.3 
Total Sample 56 17.2 17.8 / 54 16.3 16.9 16.9 


* The mean scores shown in Table 3.3.4 are based on data from pupils who responded at all time points (listwise deletion of 
missing values). Please see Appendix 7 for more detail including the original (raw) versions of the variables. 


3.3.5 Costs 


The following cost information has been provided by the ThinkForward Team. 
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Table 3.3.5a: Cost of pilot study 


Cost of pilot study 


TOTAL Per pupil 

aaa a £10000 e16 
Recruitment costs £2,025 £34 
Training £1,620 £27 
IT and phones £5,313 £89 
misc £1,782 £30 
HR, finance and payroll costs £2,965 £49 

Progression Coach salaries £235,315 £3,922 
Delivery : ee ae for £20,000 £333 
Contribution towards programme £36.267 £604 

development and evaluation : 

Management costs £24,212 £404 

Total £339,499 £5,658 


The cost of the pilot over the two years was £5,658 per pupil. This figure includes the cost of 
recruitment, a contribution towards the development of the programme, and payment of conirol 
schools. The cost of delivery over the pilot was £2426.50 per pupil per year. Since the completion of 
the pilot, the costs of ThinkForward have changed, with current costs reflected in table 3.3.5b. 


Table 3.3.5b: Cost of current ThinkForward intervention 


Today's costs for serving these cohorts 


TOTAL Per pupil 
Payment to control schools for c- £- 
additional costs 

Recruitment costs £2,025 £51 
Training £1,620 £41 
IT and phones £4,728 £118 

misc £760 £19 

HR, finance and payroll costs £1,265 £32 

Progression Coach salaries £100,401 £2,510 

Delivery i Sanne salle for £8,533 £213 
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Contribution towards programme 


development and evaluation ela ees £208 
Management costs £10,330 £258 
Total £140,310 £3,508 
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4. Process evaluation: Main Findings 


This section focuses on the findings from our qualitative exploration of the implementation and process of 
delivering ThinkForward in two schools in trial conditions. It is supplemented by interview data from the 
school lead in one control school. The section is purposively selective as the intention is to present data that 
helps illuminate the aims of the evaluation (to evaluate the extent to which coaches were successful in their 
aim of supporting young people to make a successful transition into adulthood) and our key research 
questions which are focused on the risks of spill over and other threats to programme fidelity. The findings in 
this section are also designed to enhance, explain and contextualise, through supporting evidence, the 
findings on measurable outcomes, such as attainment, reported elsewhere in the report. 


4.1 Pupil-level randomisation and the evaluation of ThinkForward 


Here we address the feasibility of pupil-level randomisation to evaluate ThinkForward as a 
coaching/mentoring programme, before exploring the ways in which the randomisation process could conflict 
with programme fidelity; firstly by looking at the risk of spill over between intervention and control groups and 
secondly by looking at potential spillover effects due to pupil-coach interactions. 


School leads reported a close match between pupils that they believed needed extra help and those the 
ThinkForward coach was already working with as a result of the randomised selection process. However the 
school Lead at school B, when interviewed during the first year of our evaluation, did report an occasional 
mismatch: 


| don't have a problem with it being [done randomly]. | suppose the only limiting factor is I've perhaps got one 
or two people that I'm thinking that are screaming out for some of [the coach's] time who | would like to be on 
[their] caseload and who is, you know [part of the] control group ...in an ideal world I'd like to be able to say 
‘This one please.’ When the lists came back there were many obvious candidates on there, there were one 
or two where we were thinking ‘Well really I've got others that would probably benefit from this.’ (School 
Lead, School B) 


It should be noted, of course, that in the absence of the ThinkForward intervention, the young people 
selected for either intervention or control cohorts would more than likely have been in receipt of some other 
form of support from the schools in question, though an exploration of this ‘business as usual’ support was 
not part of our evaluation. 


4.1.1 The potential for spill over 


One threat to the effectiveness of a scale-up trial of the ThinkForward intervention is the potential ‘spillover 
effect’ created by interactions between the within school intervention and control cohorts in the within-school 
design. This can take the form of members of the control group effectively benefitting from the same kind of 
support (e.g. behavioural support and other in-school mentoring) that the ThinkForward intervention group 
received. Members of the control group could also become more ‘self-aware’, altering their behaviour and 
attitudes in response to the experiences of their peers (who are often friends and occasionally even relatives 
- see section 2.4.8). 


The way the trial was designed made this more likely to happen, given that the control group were informed 
(for ethical reasons) of their status as a group that were deliberately excluded from receiving the intervention. 
Therefore, they knew the coach and about the ThinkForward programme, and were thus (potentially at least) 
more aware of the issues ThinkForward is designed to ameliorate than other young people. Indeed, we 
considered at the outset that they may exhibit behaviours and attitudes similar to the intervention group and 
attract other types of intervention - including detentions - that would bring them into closer contact with the 
intervention group. 
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4.1.2 Spill over in practice 


Therefore our evaluation set out to explore interaction between the two in-school cohorts and the level of 
status awareness. The coach at School A admitted that: 


When | first met with them | told them that students were selected at random, which wasn't a lie it just wasn't 
the whole truth...And then as | started building those relationships and we started going through their initial 
action plans | then kind of gave them the background to it and explained the reasons behind it. (Coach A, 
Interview 1) 


This can lead to resentment, as happened at School A during the first year of the evaluation (2014) when the 
coach had to defend the selection decision to young people on the basis that the cohort list could be altered 
if necessary in the future (Coach A, Interview 1). The alternative - to offer some level of support to the control 
group - would naturally (though understandable from the ThinkForward coach's point of view) lead to some 
degree of trial contamination: 


| have to be ...quite kind of strict ...1 have to be mindful of the fact that | want my students to do better than 
the ones in the control group ...if students are coming up to me asking me for help they could be in that 
control group and | could therefore be supporting them and pushing them forward and actually damaging the 
results and damaging... the good work that's being done. (Coach A, Interview 1) 


This evidence of self-awareness of the risks of spill over by the coach can be seen as an example of 
attempting to avoid structurally determined or constructed spill over (as opposed to spill over caused by 
more indirect relationships between friends and family). The extent of awareness of young people's status 
within the trial was explored in focus groups with control group pupils. 


4.1.3 Control pupils' awareness of intervention 


Focus groups with control group members in both schools revealed that some knew they were in a control 
group and some did not. Those who were aware of being in the control group were not sure of the basis for 
their selection; one pupil Suggested that the control group were perhaps ‘less problematic’ than the 
intervention group, Suggesting that they were only partially aware of the rationale for selection. In School B 
one control group pupil had asked the coach to be allowed in the intervention group as it was seen as 
beneficial; another praised the work the coach had done with his brother: 


[The coach] works with young people trying to help them with CVs and that basically. My brother is in the 
group. [The coach is] like brilliant with him and knows what he likes and is committed, like inspired him to do 
stuff as well on a level that he never thought that he would be able to do. (Male Y11, School B focus group) 


There were, however, mixed feelings from control pupils about whether it was beneficial to have worked with 
a coach: 


Would be good, they support you, they would look out for you (Male Y11, School B focus group) 


You get help in class anyway and if you don’t do it then you get a teacher shouting at you or crying anyway 
so it would just be the same, just another person telling you that you need to do this... but calmer. (Male 
Y11, School B focus group) 


Conflicts between the needs of the schools and requirements of the trial 


Clearly, for the purposes of the trial, coaches should only work with the intervention group and avoid any 
contact with other young people that require various types of support. However, from the school's 
perspective, the ThinkForward coach is but one of often many ‘pastoral’ support workers and it is normal 
practice for school-wide consideration of wider issues. In practice a series of negotiations on who needs 
support and who is best placed to offer it happens on a regular basis, as noted by one of the coaches: 
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So in my first few days | made sure that | checked in with the inclusion department, the mentors and the 
CSO Child Protection and Safeguarding Officer. | made sure to check in with her and make sure that she 
knows the young people that are on my cohort so that we're sharing information. And then the Year 11 and 
Year 10 mentors as well, | clicked in with them and Pupil Support and Pupil Referral which are where some 
of my kids are quite likely to be quite a bit of the time. (Coach B, Interview 1) 


This inevitably leads to internal pressures to compromise the trial conditions and threaten spill over. 
To ameliorate or limit the potential for spill over the coach had to be selective in the support offered: 


| said look | can do the period 2 because my students are in there, but | can’t do the period 4 because none 
of my guys are in there and | can’t use my time in that way,... so | make sure, you know | do what | can, but if 
my young people aren’t involved, | have to turn around and say, look it’s not my remit unfortunately. (Coach 
A, Interview 2) 


Another example of this careful treading of the line between the intervention group and other young people 
was expressed by the Coach at School B: 


We have action and concern meetings each week - all the relevant members of staff in attendance, the 
upper/lower school head, the behavioural manager, the deputy head, any other relevant support staff, we 
highlight young people in the year group who are a concern due to behaviour issues, child protection etc, 
mine are often on the list, but there are other young people. (Coach B, Interview 2) 


In summary, these findings demonstrate that interventions such as ThinkForward operate in a context of 
overlapping school responsibilities and this makes complete separation of cohorts - and of extrapolating 
effects - difficult in practice. 


4. 2 Randomisation spill over due to relationships between TF coaches and pupils 


Initially, coaches were employed for each intervention school for the autumn term 2013 after intensive 
training including the use of the ThinkForward tools and Sharepoint system for recording progress. The 
coach in one school left very early in the process and was not replaced until February 2014. The new coach 
was not able to access the required training until summer 2014; however previous work history had enabled 
a smooth transition into the role. 


4.2.1 Background and training of coaches 


Both ThinkForward coaches involved from February 2014 onwards had experience of working in school 
environments. The coach in school A had a background in youth offending casework and running youth 
inclusion projects for a London Youth Offending Team. This coach also had a background in IT and was 
comfortable with the ThinkForward tools and processes. The coach in school B (coach B) also has a 
background ‘working with young people, mostly in creative settings like drama schools and clubs’. Coach B 
had worked towards a certificate in counselling and as part of that developed a music therapy project which 
was piloted in another ThinkForward school (not involved in the trial). Both coaches noted the extensive 
selection process that involved group work sessions which benefitted both coaches. Coach B didn’t get the 
initial ThinkForward coaching training and reported that this was likely due to having already worked for them 
alongside a colleague and so would be able to ‘hit the ground running’. Coach B however did receive training 
in the systems that would be used. 


Initially, coaches met with the pupils in their cohorts either individually or in small groups to introduce 
themselves and to hand out consent forms. There were some issues where certain pupils identified were off 
site, had very low attendance or were not interested in taking part in the first instance. Once rapport had 
been established with the young people, the coaches seemed popular in both school settings which appears 
to reflect positively on ThinkForward training and the selection of coaches. 
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One-to-one interventions such as ThinkForward necessarily rely on being responsive or reactive to the 
needs of the intervention group and in the two intervention schools this was reflected often in the choice of 
outside-school activities: 


We have done outdoors in the city adventure, obstacle course, and this high jump thing, they run a 
very good programme in terms of team building and reflecting on what you have learned. Business 
mentoring, and some more casual things like: bowling, cinema, dinner, | always get the young 
people to choose it and organise it so it empowers them to do the research and do the bookings.’ 
(Coach B, Interview 2) 


The idea of extra-curricular activities is to widen young people's perspectives, to offer an opportunity to 
identify role models and possible alternative futures. Both coaches arranged trips outside of the school, with 
the coach in school A, for example, taking pupils to an adventure activity day and the coach in School B 
taking pupils to the theatre among other trips. ThinkForward is also reliant on the coaches building and 
maintaining good relationships with the parents of the intervention group. This can begin when young people 
are selected for the intervention. Even when the young people are engaged, parental relationships are 
important to secure their involvement in activities: 


I've had to chase up a few parents in terms of getting enrolment forms signed, there's been some 
language barriers as well, obviously there's been certain parents where the young person's said 
‘Can you phone my mum, she doesn't understand.’ Then when | organised this last minute theatre 
trip | had to drive around, because | had 24 hours to sort it out, | had to write the consent letter and 
go and get it physically signed by the parents, because | knew there's no way | can do this unless | 
physically get those signatures, | wouldn't be able to. So | met a few of them that way. (Coach A, 
Interview 1) 


The nature of interactions - face-to-face versus group work approaches 


The evaluation explored some issues related to differential responses to group work and one to one work. 
Many of the young people felt that one to one support from the coach had helped with their confidence. 
Group work was preferred by other young people as it enabled them to realise that some issues are general 
rather than (just) personal: 


Because you've got your own worries and then you've got the group to kind of extend it to a certain 
level. I'd say like working in a group it just, it's more informative and maybe the questions that they 
raise might have an impact on you. (Male Y11, School A focus group) 


Equally importantly, raising confidence though this kind of support increases the chances that young people 
will take up other supportive interventions offered by the school: 


And since then I've been going, like with my revision classes on Saturdays and [the coach is] telling 
me | should go and everything and they actually really helped me with my exams that | was doing. 
(Female Y11, School A focus group) 


In contrast, at School B one young person in Y11 noted that he had struggled with behaviour issues related 
to a lack of confidence: 


[The coach has] helped me with my confidence, | think if it wasn’t for [the coach] | would not be here 
now, | was pretty bad, [coach B] has helped me change my behaviour (Male Y11 School B focus 


group). 


This was a common message: 
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Everyone sort of has problems, everyone goes to [the coach] first and [the coach] sort of helps when 
it comes to us applying for colleges and helping us so we can leave school. (Male Y11 School A 
focus group) 


| think it is a good idea as in, because | get angry a lot in school and | have a lot of arguments, sort 
of fights and | have a one to one with [the coach] and it helps, it really helps. So when the teachers 
see me angry, oh you could get [coach A]... and then we talk and then | go back to lesson and I’m 
fine. (Female Y11 School A focus group) 


This indicates that a combination of one to one and group work may be appropriate to meet the needs of 
different young people. This in turn may be problematic for the fidelity of the intervention in trial conditions, 
because the degree of replicability is necessarily contingent on coaches in other school contexts being able 
to flexibly respond to young people's needs. To aid fidelity, ThinkForward provides a Coach Handbook with 
an established methodology, whereby coaches use approaches and resources and tailor them to the needs 
of the school or young person. 


4.3 Impact of the TF programme on measurable outcomes 


The process evaluation did not directly gather evidence about the impact of the ThinkForward programme on 
attainment or other measurable outcomes - however we were able to identify certain circumstances that 
impacted the ability of coaches to operate in the way prescribed by ThinkForward. In turn these are 
considered below: relationships between coaches and their School Leads’ (including negotiated access to 
pupils); the nature of ThinkForward processes. 


4.3.1 Relationship with School leads 


Both schools provided a Lead to work with and support the coaches as was part of the ThinkForward 
specification. In school B the school provided two; the assistant head teacher and the assistant to the head 
of Year 11 (a careers teacher). The coach worked on a day to day basis with the careers teacher and had 
meetings with the assistant head on a monthly basis. This combination was said to be working well: 


[The lead teacher] is on board with what | am trying to do here but he is extremely busy. But having 
the deputy head backing it is extremely useful for when | need to get things done. (Coach, School! B) 


In School A the coach worked with only one Lead; in the first year the Lead was an assistant vice-principal, 
in the second a vice-principal. 


Overall, Leads at both schools were positive about the ThinkForward programme. There was some caution 
in one school due to being unfamiliar with the programme and being keen not to disrupt pupil's workload in 
Y11 particularly. This lack of access to young people in Y11 may be linked to the lower levels of 
improvement seen by this cohort in relation to ThinkForward’s Mind Sets relative to cohort 2. Leads at both 
the schools felt that ThinkForward fitted in well with the school's current priorities. Having a coach in the 
school reinforced some of the intervention work they were already doing, including a mentoring scheme for 
targeted pupils, but also enabled school leads to be ‘freed up’ to work with other students who were not on 
the coaches list (including potentially some of the control group - another example of spill over). 


Coaches in both schools reported having positive relationships with the school leads in their school. 
Reasons given for these good relationships were that the school leads could see the value of the 
ThinkForward programme, and were supportive to the coaches in making them feel welcome and a part of 
the school. The benefits of the coach in school from the school leads' perspective were said to be having 
someone from a different background to work with and relate to the young people and ‘an extra pair of 
hands’. School leads in School A also hoped it would open up opportunities to share information with other 
schools involved. 
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4.3.2 Access to pupils in appropriate conditions 


The coaches based in the two intervention schools had differing experiences in regards to their working 
conditions. Coach A had been given an appropriately quiet office location in which to work with pupils in the 
first year, and felt it was working well; however in the second year the office arrangements changed and the 
coach was sharing with two different members of staff. The coach in School B had struggled from the outset 
with available ‘private’ space to work in. Pupils were not inclined to come and visit the coach at this office, 
shared with an Education Welfare Officer. Both coaches occasionally found it very difficult to find available 
rooms to work with pupils one to one. 


As might be expected with the run up to GCESs, coaches found it difficult to get access to Y11 pupils. 
Coaches reported struggling to find time to work with Y11s as they were not able to be taken out of what is 
considered ‘protected curriculum time’ (most sessions apart from PE) so a degree of negotiation had to be 
entered into. Coaches routinely had to ask pupils to stay behind after school or come to see them during 
break, lunchtime and in some cases detentions. One coach reported that this made it difficult initially to meet 
targets for ThinkForward, for example completing the action plans for each student. As time went on 
coaches began to find ways to work with pupils, one coach reported however this was still on an ad hoc 
basis, as and when pupils were prepared to come in. The other coach discussed how they had been going 
into pupil's maths and IT lessons to support them with these subjects. In the Coach's opinion the Y10 cohort 
had been easier to work with as there were fewer ‘protected curriculum’ restrictions on their time. 


Clearly, given the nature of coaches' being embedded in schools, reproducibility of conditions of working and 
access to young people (essential for trial fidelity) are always going to be difficult to guarantee. 


4.4 ThinkForward processes and measurable outcomes 


As indicated in Section 2.2, the ThinkForward process involved a quite tightly structured and relatively 
complex set of processes involving a combination of online and other systems, with both positive and 
negative responses emerging in relation to aspects of these processes. 


At the time of the first interview the coach at School B had not yet become fully familiar with the 
ThinkForward tools and processes due to a late start (replacement for the original School B coach 
who had left). Coaches used both online data systems and paper files to log contact with young 
people: "Reports can be pulled off to see how much contact time, what interventions, exactly what 
each young person has had". (Coach B, Interview 2). 


One coach talked about using the ThinkForward SharePoint site as a way to record work readiness and that 
this had been a useful tool for meetings to assess pupils and for pupils to self asses where they were: 


| feel there’s lots in place to be able to measure where they’re at and the database has been 
improved so that we can actually monitor, we can actually score them on the different capabilities for 
each session. (Coach A, Interview 2) 


The Personal Development tracker, (later renamed the Work Readiness Tracker) used to record pupils mind- 
sets, was thought to be valuable by coaches to an extent, although the coach at School A felt that the 
system was not user friendly and that it took excessive time to enter pupil data. Both coaches also spoke of 
the difficulty in getting pupils to meaningfully participate in the process, with only half the pupils in school A 
completing it ina way deemed to be appropriate: 


Some of them have openly said ‘I had fifteen minutes before | was leaving school, | didn’t take it 
seriously, | just ticked what | wanted to tick, what was the closest answer to me.' (Coach A, Interview 


1) 


However, coaches felt that the tracker was useful for planning sessions and interventions with pupils if it was 
completed correctly. 


Education Endowment Foundation 50 


ThinkForward 


As indicated in Section 2.2, coaches were asked to complete detailed action plans for each of the pupils in 
their cohort. One coach found the action plan to be a useful tool for their own planning, but perhaps not for 
the pupils as they were unlikely to look at their plans again. The other coach found the ThinkForward ‘wheel 
chart’ to be of most use, saying that it covers aspects of pupils lives and helps them to plot how ‘at risk' each 
individual pupils may be in any one area of their lives so that they could plan any necessary intervention 
work accordingly. 


Adherence to ThinkForward processes is potentially the area where fidelity can be more easily demonstrated 
as it can be captured through interactions with the intervention tools. However given the qualitative nature of 
our interviews it was not possible to make any systematic evaluation of the extent to which these were being 
used by coaches. 


4.5 Producing better outcomes: the perceived impact of ThinkForward 


At the time of the first interview (four months into the intervention) in School B, it was felt by respondents it 
was too early to make judgements about the impacts of the ThinkForward coach. However school leads 
were impressed with how well the coach had managed to foster positive relationships with the intervention 
pupils. One school lead noted that, for the Y11 cohort, it was unlikely that there would be time to do much 
more than to keep these relationships going, offering a supportive role to these students before they left 
school. 


In the second set of interviews (one year and five months into the intervention), coaches and school leads 
were able to see the impact of coach's work with pupils, particularly in terms of behaviour. The level of 
impact was perceived to be based on how engaged each young person had been with the ThinkForward 
process, and some pupils were reported to be much more engaged than others (with the latter group a 
minority). For those pupils who had bought into the process and worked closely with the coach, there had 
been some notable improvements as described by the coaches: 


A couple of them have completely turned things around since | started working with them, their 
behaviour was awful when | came into school, | worked closely with them and they have completely 
turned things around, but there are other factors, other people working really hard and the young 
person themselves is the one to make the change but yeah | have had an impact. (Coach B, 
Interview 2) 


This suggests a number of reasons why pupils may have improved in this area; however the coach felt that 
pupils having a ‘constant adult’ in their lives whom they could trust had been key to their positive changes. 
One of the school leads commented on how pupils in the intervention cohort had changed and how they now 
interacted with each other in a positive way, and put this down to the 'deep mentoring approach’ which mixed 
working in groups, one to ones and activities/trips out of school. This school lead felt that the programme 
gave ‘real life experiences’ and helped pupils to aspire more. 


Coaches had also worked hard with pupils on their future aspirations, noting that some of the pupils they 
worked with had not valued education previously and so needed support to think about their future options. 
One school lead strongly praised the work of the coach in his school: 


| believe [the coach has] got all of them except for one or two, have put in at least two applications 
for college, fantastic, | mean that just wouldn't have happened without [the coach] (School Lead, 
School A). 


Intervention pupils were asked about the impacts of being involved in the ThinkForward programme, with 
most comments centred on an improvement in their confidence and self-belief, particularly where the coach 
had helped them to work through personal problems both at home and at school: 
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With [pupil name] before he used to be a really shy person, he sat at the back of the class never 
talking to anyone, now he will ask the teacher for help which he never used to do. (Male Y11, School 
B) 


| didn't think | could do it because of the way | am, but with ThinkForward they kind of say 'You can 
do anything if you just put your mind to it’. (Female, Y11, School A) 


There were also positive comments on planning, communication skills and future aspirations: 


We learnt about different interview skills and working within a business... that was really helpful 
because | used some of the interview skills in my college interviews and I've got into like two out of 
three. (Male, Y11, School A) 


A number of pupils across both schools explained how their coach had supported them with choosing 
subjects to study and colleges, writing personal statements or supported them in gaining work experience. 


| feel [the coach] is very supportive, | was lost but [the coach] guided me with what college | should 
choose and filling in my personal statement. (Male Y11, School B) 


One pupil in school A described how a ThinkForward presentation in their school had opened his eyes to 
possibilities of job prospects after school that he had been unaware of. Pupils also made more general 
positive comments on the coach with pupils in school B describing the coach as ‘like a friend’ and like a 
parent to them. 


It is clear that the process evaluation evidence presents a more positive picture of ThinkForward's impact 
than does the trial findings. This is not surprising given the qualitative nature of exploration and the highly 
individualised - and sometimes short-term - impacts that this kind of intervention can have on young people. 
In addition the fieldwork was conducted with small numbers of interviewees who were sometimes necessarily 
more likely to be positive, such as self-selecting young people who attended focus groups. 


However, the ability of coaches to secure and maintain relationships with parents, who may potentially not 
want their child to be identified with an intervention sometimes portrayed to funders and corporate sponsors, 
as those ‘at risk’ of failure, is also important to its success. The coach at School A reported on the potential 
conflicting messages between ThinkForward as a positive supporting intervention for individuals and the 
corporate image that portrays it is a way of fixing social ‘problems’, citing a mother who: 


...was really not happy with her son being identified as being at risk of NEET. | think there’s 
something to be learnt by our literature on the website, it’s aimed towards | think the corporate 
organisations and the people who we want funding from, so it’s kind of really tugging on the 
heartstrings and actually parents maybe don't want to see it that way.... Like | say the ThinkForward 
in Action is the hard sell, it’s the tugging on the heartstrings for the corporates. You know there’s 
something uncomfortable and for me in the way it’s kind of dressed up and put to them. (Coach, 
School A) 


4.6 Summary: evidence of promise and feasibility of approach from the process 
evaluation 


This section has shown that there is evidence of perceived promise that the programme can ameliorate the 
risks of young people becoming NEET as evidenced by the positive relationships coaches were able to 
develop with the young people in the intervention groups, engagement in group and individual activities, and 
the positive perceptions of the impact identified by school leads, coaches and young people in the 
intervention group. 


Despite the perceptions of positive impact, it is difficult to disentangle the influence of the intervention 
specifically. There are a number of factors at play which may affect the young people involved; for example 
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changes in relationships between pupils and teachers, other interventions happening in the school, or as one 
school lead pointed out, a pupil moving into Year 11 and deciding to work harder in order to gain the 
necessary GCSEs: 


It’s very hard when you're evaluating why that child is so successful to say, well ThinkForward was 
50% of that success and teacher x was 10% of it, and to quantify that, and it becomes a bit more of a 
subjective judgement. (School Lead, School A) 


There is also the issue of attributing the impact of the coach as an individual, and the ThinkForward package 
in its entirety. The two coaches had experience working with young people in similar roles and were clearly 
both passionate and motivated individuals. Comments on impact, particularly from pupils, but also from 
coaches themselves, seem to relate to the coach rather than specifics of the ThinkForward ‘brand’ of 
coaching. 


However, the ThinkForward approach, which combines one-to-one and group work is clearly, at least in the 
opinion of young people and School Leads well designed and effective as an intervention in the opinion of 
young people and School Leads. The emphasis on group working and the school-driven requirement for 
ThinkForward coaches to work collaboratively with others with pastoral roles does represent a risk of spill 
over. Although spill over in school would usually be a positive outcome as there is a potential to help more 
young people, in the within-school trial it is detrimental due to fidelity being dependent on a denial of any 
benefits of the programme being detected by those in a control group. 


4.7 Readiness for Trial 


Given that ThinkForward significantly revised its selection procedure (reverting to a focus on beginning with 
Y9 and having smaller cohorts of Y10 to enable deeper and more parent-focused work for the older age 
groups), the process by which cohorts were randomly selected for this trial was sub-optimal and likely to 
have contributed to its relative lack of impact. This research suggests working conditions and access to 
pupils are often not ideal for the purposes of a within-school trial and therefore ThinkForward in this form is 
not yet suitable for trial. Conditions of access in ThinkForward schools need to be more controllable and 
broadly comparable in order to create the conditions for an effective within-school trial, however it does not 
follow that all schools adopting ThinkForward would have to organise their affairs identically for the coaching 
intervention to provide benefits for pupils. The process evaluation was not designed to find ‘hard’ evidence of 
impacts on attainment, however there was 'soft' evidence in the form of a number of positive reports from 
school leads, coaches and pupils themselves about impacts on their behaviour, motivation and future 
aspirations. The conditions of access and environment are important to the effectiveness of the intervention 
and to a trial which has to be based on the degree of comparability that is inherent in the intervention design. 


With only two intervention schools in the trial, it is impossible to be conclusive. Findings from the process 
evaluation highlighted the difficulties of disentangle the positive impacts of individual coaches and the 
ThinkForward intervention itself. Comments on impact, particularly from pupils seem to relate to the 
individual coach rather than specifics of the ThinkForward intervention. ThinkForward coaches and school 
Lead contacts were more likely than pupils to attribute positive outcomes of the intervention to the design of 
the intervention and in particular the selection and ongoing training of coaches. 
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5.1 Limitations 


The pilot was small scale and limited to four London secondary schools and the findings should not be 
considered as generalisable beyond these schools. 


A second limitation relates to the problems with randomisation explained in section 2.3 that resulted in a 
baseline imbalance where, on average, pupils in the intervention group had a higher predicted NEET score 
than compared with pupils in the control group. This is likely to have resulted in understating the any positive 
impact of ThinkForward across the outcomes. As specified in section 2.4.7, models included covariates to try 
to address the observed baseline imbalance but this approach is unlikely to address the potential 
unobserved imbalance associated with a relatively higher NEET risk. 


A third limitation relates to spill over. Evidence of spill over came from both the quantitative analyses and 
process evaluation. This suggests that the use of pupil-level randomisation also risks understating the 
positive impact of ThinkForward across the outcome areas. 


A fourth limitation relates to the low response rates for surveys 2 and 3 which bring issues of non-response 
bias for analyses of secondary outcomes collected via the longitudinal surveys 


A fifth limitation relates to the involvement of coaches in facilitating data collection for the surveys and 
arranging interviews for the process evaluation. 


5.2 Interpretation 


Little / no evidence of impact was found across the four outcome areas for this the pilot. No evidence of 
impact was found for the primary outcome KS4/GCSE attainment. Where evidence of 'impact' was observed 
it was predominantly located within the first six months of the trial and so represents ‘impact’ in the short 
term. Evidence of longer term impact was outside the scope of this pilot.. 


It should be acknowledged that the limitations noted above undermine the validity of drawing strong 
conclusions from the impact analyses for this pilot. A stronger methodology with a better baseline balance 
and higher survey response may have led to evidence of impact but from what we have analysed, the 
quantitative evidence of impact is scant. The findings from the process evaluation are more consistently 
positive from coaches, school leaders and the participating pupils. However, in this pilot we found no 
evidence that this resulted in a measurable impact amongst participating pupils relating to KS4/GCSE 
attainment, unauthorised absences, future educational expectations and ThinkForward mind sets. 


5.3 Reflections on trial methodology 


The two main aims of this pilot were methodological. Whilst evidence of promise specific to ThinkForward 
was not found, a number of methodological issues did emerge that will need to be addressed in future trials 
of similar interventions. We reflect on four methodological areas: 1) randomisation and the ThinkForward 
eligibility criteria, 2) approaches to data collection; 3) friends and family links between the intervention and 
pupil-level control groups; and 4) the underlying two-armed RCT methodology. 


In terms of randomisation and the ThinkForward eligibility criteria, a standardised approach was used to 
select pupils for the ThinkForward programme. The criterion was standardised across the four London 
schools using a scoring mechanism designed by ThinkForward. The scoring mechanism generated a 
predicted NEET risk score. The approach usually taken by ThinkForward was to consider the relative 
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predicted NEET risk of pupils within each school they worked with? and then interview pupils with a 
relatively high score to assess whether or not they might benefit from the programme. The standardised 
eligibility criteria we used led to problems with the pupil-level randomisation of this trial that led to the 
baseline imbalance discussed in section 2.3. To summarise, the standardised criteria resulted in different 
numbers of ‘qualifying’ pupils in the two intervention schools, a fluctuation of pupil numbers across the two 
year groups within the two intervention schools, whilst the number of pupils within the intervention group was 
fixed at 20 per year group. This was done to try to address a request by ThinkForward to work with pupils 
deemed to be most at risk. This led to a baseline imbalance where on average the intervention group had a 
higher predicted NEET risk score and lower KS3 attainment compared with the pupil-level control group. The 
intervention group were also less likely to be 'FSM', less likely to be female and more likely to have a SEN 
with a statement or School Action Plus when compared with the pupil-level control group. In this pilot, we 
addressed the observed baseline imbalance through the use of baseline covariates and difference in 
difference regression modelling, but future trials will need to resolve issues around eligibility and develop 
standardised criteria that result in randomisation that produces a better baseline balance. 


With hindsight, it would have been preferable to have allowed for a longer lead in order to allow all initial 
stages to be completed before commencing any of the random allocation. This would have provided time to 
develop a randomisation scheme that fitted the needs of ThinkForward but did not result in a baseline 
imbalance and to improve on the ‘water-tight’ robustness in the research design. Specifically, it should be 
noted that whist school and pupil level randomisation was structured to ensure that they both took place after 
the MoU was signed and following two stages of opt-out consent and baseline data were collected, in reality 
it was some months after randomisation before the baseline survey was transferred from questionnaires into 
data files for analysis and scrutiny. It would be preferable to have obtained and processed all of the baseline 
questionnaire data prior to randomisation taking place. 


However, the introduction of changes to the way a programme such as ThinkForward select eligible pupils 
will have reduced the (ecological) validity of our evaluation design. The evaluation was centred on an RCT- 
adapted version of the programme rather than the programme as it was originally devised. This may reflect 
how further development is needed before ThinkForward is suited to an RCT centred evaluation. Whilst 
standardisation of eligibility criteria brings advantages with respect to an RCT methodology, a design that 
allowed some school-level variation around eligibility would lead to an evaluation more closely aligned to the 
intervention under evaluation. Providing that this eligibility selection took place prior to randomisation and 
school-level variation was acknowledged within the analysis plan, this approach would not result in 
introducing bias into the design. 


We feel that there are potential outcome areas should be considered for inclusion as outcomes in any future 
evaluation of ThinkForward. In particular, detail on post-16 educational experiences and post-16 
expectations and experiences around employment and training. Indeed, support over the following two years 
(Y12 and Y13) when the young people have left school and gone into further education or employment with 
training is integral to the ThinkForward model, albeit not the focus of the trial which was restricted to the last 
two compulsory school years. 


In terms of approaches to data collection, the use of coaches to facilitate data collection must be addressed 
in future trials. Future research designs should ensure that coaches do not access survey questionnaires 
during the trial. One approach might be to explore the use of an online survey and randomly assigned 
invigilators to oversee data collection. Whilst an online approach is considered more risky in terms of 
response, given the Y10 and Y11 age groups within this pilot, it may have been a better option. This would 
have also brought the benefit of helping to detach the coach from the data collection process which would 
result in a more robust research design that would be less open to the risk of subjective and objective bias 
arising from the coach involvement. 


22 Selecting the 10 or 20 pupils with the highest predicted NEET score for example. 
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In terms of friend and family links, whilst we argue that collecting friends and family details directly from 
pupils is a more valid and reliable approach to collecting this it indirectly (for example, having a teacher to do 
this) there are some aspects we feel could be improved on. These include keeping the details on friends and 
family members separate within the analyses”°; collecting this detail longitudinally (to acknowledge possible 
fluidity over time of friendship networks) and including details on the numbers of friends/family members 
identified (overall and the links between the intervention and pupil-level control group). 


Finally, we call into question the use of a two armed trial methodology to evaluate one to one or small group 
focused coaching/mentoring interventions such as ThinkForward. Whilst, with some methodological 
development, a two armed trial might provide reliable estimates of the statistical effect of the coaching / 
mentoring experience across a range of quantifiable outcomes, the ability to validly attribute the cause of any 
observed effect to the specific programme is more questionable. A two armed design would not be able to 
assess whether cause is more attributable to the professional experiences, skills and understanding of the 
coach; to the coaching programme that is followed by the coach or to a combination of the two. A three- 
armed trial and development of a placebo group (Such as coaches who do not follow the specified 
programme) would be one way to trying to address this but would clearly result in a substantial cost 
increase. 


23 The baseline survey collected details on whether control group pupils had friends within the intervention group 
separately from whether they had family members within the intervention group. For simplicity, these two (friends and 
family) were combined for our analyses. Within a larger scale trial, it may be valuable to look at evidence of spill over 
through ‘friendship’ networks separately from spill over through family links. 
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Appendix 1: Pupil / Parent Opt Out Consent Form 1: 


This form was given out to all Y10 and Y11 pupils in all 4 schools prior to school-level 
randomisation and sought to gain (opt-out) consent for participation in the surveys and to 
link this data to NPD data. 


Dear parent/carer, 


Your child's school is taking part in a trial initiative designed to help young people in their 
transition from school into work. 


We wish to conduct a survey of all pupils in your child's year at school. Responses from 
the survey will be matched with the National Pupil Database and shared with academics at 
Sheffield Hallam and Essex Universities, the EEF and the UK Data Service for research 
purposes. 

Rest assured no individual child will be identified by name in any reporting. All responses 
are confidential. We will anonymise your child's name by using an identifier number, in 
security protected computer files. EEF may use the identifier number in the future to link 
data from this project to data that is routinely collected on pupils by the government, for 
example exam results. 


This survey is important because it will help us understand the potential benefits of an 
initiative designed to raise young people’s attainment and engagement with school. 


If you would prefer your child NOT to take part in the survey, please complete the slip 


below and give it to your child's teacher/ take it to the school office. If you would like more 
information, please contact us using the details below: 


Please return this slip to your child's teacher (or school office). 
ThinkForward pilot evaluation 
| do not give my permission for my child to take part in the survey 
CHILES TOU VANINGS ctcetnscttuars hanna nateansiditonsnnccdeamezaumardtcenadaatnstsainnade om ieaeereaetuananacescaeesaens 


SIOMEUS ohragavascaenbnens etsceacyaaueeseesshoeni aneebasieasa at bengeoans ohpheasueeaa a bebsaacsW aneattdanwedeasaweastas 
Parent/carer 


AUS esc eee eres ect oe aces ee ctchaane cheteatdcetne sess cautoes tenes ceantoncacdiace suns tuautetsenweranctaettunmutece 
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Appendix 2: Pupil / Parent Opt Out Consent Form 2: 


This form was given out to all Y10 and Y11 pupils in the 2 intervention schools prior to 
pupil-level randomisation and sought to gain (opt-out) consent for participation in the pilot 
trial. 


Dear parent/carer, 


Your child's school is taking part in a trial programme called ThinkForward, designed to help young people to discover 
and achieve their ambitions. ThinkForward places inspirational coaches in schools, to work with young people from Year 
10 onwards and provide them with targeted and intensive support through school and into their next destination. 


Some children in your child's school will be selected to get the ThinkForward programme from Summer 2013. Pupils will 
be selected for the ThinkForward Programme randomly (similar to a lottery). 


The reason for this is so that we can evaluate ThinkForward by comparing pupils who experience the coaching with 
those who do not. The specific focus will be in identifying any educational and / or employment benefits from participating 
in the programme. 


This letter is to let you know about the evaluation. If you would prefer your child to NOT take part could you please 
complete the slip below and return it to your child's teacher (or the school office). If you would like more information, 
please contact us using the details on the next page. 


Once we have allowed two weeks for parents and carers to opt out of this evaluation, we will then randomly select pupils 
to get the ThinkForward programme. You will receive a letter to let you know that your child will be receiving the 
programme IF they are one of the randomly selected pupils. 


The project is being paid for by a government grant. You can read more about the 'ThinkForward' programme on the 
next page. Please read this information and talk to your child about the project. 


If you do NOT want your child to take part in the project, please fill in the slip below and return it to your child's 


teacher/school office by [Date]. If you do not return the slip by this date, we will assume that you are happy for your 
child to take part. 


Please return this slip to your child's teacher (or school office) 
ThinkForward pilot evaluation 
| have read the information sheet and understand what is involved for my child in taking part in this evaluation. 
| do NOT give my permission for my child to take part in the ThinkForward evaluation 
CRS TUN MEIN SF reesces sre sad tndentitetadcracemnneakatcanaaunigeambacy niente eugene tian nee eter edn meienateees 


SIQHMGG 3 asi bedetgacccnchs ale vrnmscddiad dddngarntiahbensubhtnaanbedan gaan adeatianuanredbleiernmunsnes noumeenambatiarenbat Parent/carer 
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ThinkForward evaluation 
Who is running the project? 


ThinkForward is a programme run by the Private Equity Foundation (PEF) with a charity called Tomorrow’s People, which 
provides highly trained Coaches to work with some 14 year olds as they progress through to GCSEs and post-16 choices, 
supporting them to make a successful transition into work. Schools identify the young people at the end of Year 9 who could 
benefit from help to reach their full potential at school. Coaches then support those individuals through their school journey and 
into a job that best matches their skills, personality and ambitions. 


Whilst ThinkForward Coaches will carry out the work with young people in school, researchers from Sheffield Hallam University 
and the University of Essex will carry out an independent evaluation of the programme to ensure that it works to the benefit of 
young people. 


What does the evaluation involve? 


Pupils selected to benefit from ThinkForward this year (2013) will meet a Coach who will help them complete an online 
questionnaire to identify their specific needs. They will also be asked to complete a survey designed by the university researchers 
which asks them about what they hope to gain from education, for example what sort of jobs or careers they wish to go on to, do 
they think about further and higher (university level) education and who or what influences their choices. Children not selected for 
ThinkForward in 2013 will also be asked to complete the survey but will not have a Coach in 2013. . 


What information will you collect about my child? 


Before the project starts your child’s school will tell us about how your child is doing. Later in the project we will ask the 
government to give us some more information, such as your child's key stage results and gender. The evaluation team (from the 
university) will ask some of the children to talk in small groups about the project with a researcher who will visit their secondary 
school. 


What will you do with the information you collect about my child? 


We will use the information to report on the success of the ThinkForward programme. A short summary of our findings will be put 
on the Education Endowment Foundation website. This will help your school and other schools see how such programmes can 
help young people. The evaluators may publish the findings in an academic journal. 


No schools or children will be named in any report or publication. All information about children, including test results, will be held 
confidentially and in compliance with the Data Protection Act. Confidential data (without names) will be shared between 
ThinkForward, the evaluators and the Educational Endowment Foundation for current and future research purposes. 


What do | do if | no longer want my child to take part in the trial? 
If during the project you no longer wish your child to take part please contact the evaluation project manager (details below). 


If you have any questions please contact us using the details below. 
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PUPIL QUESTIONNAIRE 


Introduction 


ThinkForward 


This questionnaire was designed by researchers at Sheffield Hallam University 
to collect details on your views, perceptions about school, education and what 
you are planning / hoping for in your future. 


It should take between 10 and 15 minutes to complete. 


Please can you read and respond to all questions and return the completed 
questionnaire to your teacher. 


Your responses are COMPLETELY CONFIDENTIAL. No one will be able to 
identify you from the research results or the final report. 


Section A: 


To begin with, we would like to find out whether you agree or disagree with the following 


statements. There are 70 in all. 


Please read each statement and indicate using a tick or a cross whether you strongly disagree, 


disagree, are unsure, agree or strongly agree to each of them. 


There are no right or wrong answers here, we just are interested in your opinion. 


EXAMPLE: | enjoy completing surveys 
| have confidence in myself. 

| can achieve anything | want to. 

| am confident in my abilities 

| can cope with any problem 

| often worry whether | am good enough 
| make the best of bad situations 
Overall | am a lucky person 
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Strongly Disagree Unsure 
Disagree 


O0O0000 00 |0 


O 


O0O000 00 


O 


O0O0000 00 


—— ooo 
gO 
O O 
O O 
O O 
O O 
O O 
O O 
O O 
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Pend A: 


il 
i 


| believe things will work out in the end 

People describe me as a positive person 

Good things never happen to me 

| have clear goals that | want to accomplish 

I set myself goals every day 

| know what | want to achieve in life. 

| know what | would like to do when | am older 

| have no idea what | want to do with my life 

| finish whatever | start 
Setbacks never stop me 

| make sure | achieve my goals 

| enjoy working hard. 

I give up when things become difficult. 

| always change how | do things ff it is not working. 
| easily adapt to new situations. 

| am happy to change the way | do things half-way through 
| always try to see change as a positive thing 

| find changing my plans frustrating. 

| love learning. 

| seek out opportunities to learn something new. 
| always learn from my mistakes 


| am constantly looking to learn and improve 


OO0O00 00 0 0 0 0000000000000 0 
OO00 0 00 00 0000000000000 0 
O O00 0 00 00 0000000000000 0 
O00 0 0 0 0 00 0O00000000000 0 
O00 0 0 0 0 0 0 0 0 0 0000000000 0 


| do not see the point in learning new things 
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HL, it 
Drait 
tina Disagree Unsure Agree ae 
| recognise the emotions that | am feeling 
| easily recognise how someone is feeling. 
| understand what other people are thinking. 
| recognise the impact my actions have on other people.. 
| find it difficult to read other people. 
When | am under pressure | remain calm.. 
When faced with a difficult situation, | do not worry about it. 
Even when | am feeling angry | manage to stay calm 
| find it easy to stay in control of my emotions 
| find it difficult to remain calm in stressful situations 
| enjoy getting to know people 
| find it easy to develop good relationships with people 
| understand the need to build effective relationships 
| find it easy to relate to people who are different to me. 
| find it hard to build positive relationships. 
| am clear and confident when | speak to other people. 
| feel confident speaking in public 
| change how | talk depending on who | am with. 
| can persuade others of my point of view.. 
| find it difficult to persuade other people to my way of 
thinking. 
| enjoy working as part of a group 


OO0O000 00 000000000 00 00 0 0 
O00 00 000 000000o000%0 00 0 0 
O00 00 0000000000000 00 0 0 
O00 00 00 0000000 C0000 0 0 0 
O00 00 000000000800 00 0 0 0 0 


| enjoy supporting other people 
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Agree 
| aways work well in a group. O O oO O O 
lam an effective team-player Oo Oo O O O 

| find it difficult to work with other people. O O O O O 

| enjoy solving difficult problems. O O O O O 
| identify the best answer to problems O O O Oo O 
| can generate a number of solutions to problems O Oo Oo oO Oo 
| like coming up with solutions to problems. O O O O O 
As long as a problem is resolved | don't need to know the © O O O O 
root cause. 

| seek support from others when | am in difficult situations. © Oo O Oo Oo 
| always ask for help when I need it. O O O O O 
| ask for help from the right people. O O O O O 
| actively seek friends who are a positive influence O Oo O Oo O 
| make friends with people who are a bad influence Oo O O O O 
| organise my own time effectively. O O O O O 
| like to organise things. O O O O O 
| prioritise issues before deciding what action to take O O O O O 
| plan ahead to help achieve my goals Oo O O O O 
People might describe me as disorganised. O O O O O 
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Section B: Your Future 


The following statements are about your future plans and expectations. 


Please read each statement and use a score between 0 and 100 to tell us how likely you 
feel that it will happen for you. 


On a scale from 0 to 100 - where 0 would mean 'no chance of happening’ and 100 would 
mean ‘totally likely to happen’, please tell us how likely it is that you: 


Score (0 to 100) 


...Stay in full time education after finishing year 11 |] 


..apply to university to do a degree 


Now suppose that you do apply to university. On the same scale from 0 to 100, please tell 
me how likely it is that you: 


Score (0 to 100) 


..are successful in getting a place |] 
[| 


...are successful in completing the degree (graduating) 


i Page 5 of 9 || 
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Draft 
Section C: About You, your Relatives & Friends 


In this section we wil be asking for the names of you and any friends / relatives that you have 
in Year 10 or 11 of this school. 


Your answers to these (and all the other) questions are completely confidential 


The names that you provide will be anonymised in our files, no names will be stored or shared 
with anyone else (including teachers). 


Ci - About You 


Please can you write your name and date of birth in the boxes provided. 
Name (First & Last) 


pO 


Date of Birth (Day / Month / Year) 


PL LE LEE Tt 
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C2 - Family Relatives in Year 10 & Year 11 


Do you have any brothers, sisters or cousins who are in year 10 or year 11 of this school? 


Yes 


No © Please go to section C3... 


O Please can you provide details below... 


Please can you write the names of any brothers, sisters of cousins who are in Year 10 or Year 
11 of this school? - we have provided space for up to 5 family relatives. 


For each relative, can you also indicate whether they are your brother. sister or cousin? 


EXAMPLE Relative Name 
please provide first and last names 


(e.g.) Jonus Smith 


Relative Name 
(please provide first and last names) 
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School Year 
Zz Year _10 


O Year _ 11 


School Year 


O Year _10 


OYear_11 


OYear_10 


O Year_11 


O Year 10 


O Year_11 


O Year 10 


O Year 11 


O Year _10 


O Year _11 


Are they your ... 


O Brother 

O Sister 

@ cousin (Male) 
OCousin (Female) 


Are they your ... 


O Brother 

O Sister 

OCousin (Male) 
OCousin (Female) 


O Brother 

O Sister 

OCousin (Male) 
OCousin (Female) 
O Brother 

O Sister 

OCousin (Male) 
OcCousin (Female) 
O Brother 

O Sister 

OCousin (Male) 
OCousin (Female) 
O Brother 

O Sister 

O Cousin (Male) 
OCousin (Female) 
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C3 - Friends in Year 10 & Year 11 


We are interested in your close friends within this school. 


Please can you write the name of your closest friends in Year 10 and Year 11? - we have 
provided space for up to five (5) closest friends. 


EXAMPLE Close Friends Name 
please provide first and last names a Male or Female? 
. O Year_10 O Male 
(e.g.) Majury Jones Female 
fYear_11 
Close Friends Name 
lease provide first and last names Seen Waar Male or Female? 
O Year_10 OMale 
OYear 11 O Female 
O Year_10 OMale 
OYear 11 OFemale 
O Year_10 UMale 
OYear 11 O Female 
O Year_10 OMale 
O Year 11 O Female 
OYear 11 OFemale 
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C4 - Where you live 


What is the postcode for the house where you live most of the time 


PLE T TET 


If you do not know your postcode, 
please let us know by placing an *X’ || 
in this box: 


This is the end of the questionnaire: 


Thank you. 
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Appendix 4: Schools Memorandum of Understanding (MoU) 


ThinkForward 
Host School/Academy Memorandum of Understanding 


This Memorandum of Understanding dated [xxxx] is between: 
Tomorrow’s People Trust Limited School/Academy name and address 


(“TP)” 
(“The School/Academy”) 


The details of each of the officers appointed to exercise the rights and powers of the 
respective parties to this Memorandum of Understanding (MoU) are: 


Tomorrow’s People Operational Lead School/Academy Lead 
Manager 


Background and Purpose of the Memorandum of Understanding 


ThinkForward’s objective is to reduce the number of young people who are not in education, employment or training 
(NEET) in Shoreditch by 50% by 2016. We. will do this by placing inspirational Coaches in selected schools, to work with 
young people at risk of becoming NEET. Coaches will work with participants from the age of 14 and provide them with 
the targeted and intensive support to ensure that they transition successfully into work, education or training. 


The ThinkForward Initiative is the intellectual property of the Private Equity Foundation (PEF) and is delivered in 
Shoreditch in partnership with Tomorrow’s People. 


The ThinkForward initiative cannot achieve its objectives without the support of your School/Academy. We want you to 
be a key stakeholder in the development of ThinkForward. 


In particular, we would like your school to help pilot a cutting edge Randomised Control Trial together with the Education 
Endowment Foundation and its partner universities. During this stage, in order to remain scientific, the trial will use 
random allocation to select the schools in which ThinkForward operates. 


Schools will be randomly allocated to receive ThinkForward or to become a Control group school. 

Before this random allocation takes place some data collection (including a pupil survey) will take place. 

If your School/Academy is selected for ThinkForward, the whole of this MoU applies from the date of signing 

If your School/Academy is not selected (i.e.it is a Control group school), only the ‘Identify’ stage (see Section A and 
B, below) and some further evaluation activities (including follow up surveys and some interviews) will take place 
during the next two academic years. 
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e Control group schools / Academies and schools selected to receive ThinkForward will all receive a covering grant of 
[£5000??] in payment for any associated effort and cost. 
e Control group schools will be given the opportunity to receive ThinkForward at a later date 


By doing this, your School/Academy will be contributing respected research to the education sector and potentially 
informing national policy 


e The purpose of this Memorandum of Understanding is to outline key aspects of the evaluation research and how 
this relates to your school and pupils. 

e = ThinkForward within the selected School/Academies and to delineate the responsibilities of TP and the 
School/Academy. 

e This Memorandum of Understanding does not imply any legal commitment but aims to set out the common intent of 
both partners on how we will work together. 


For ALL Schools 


A) Data Collection Activities 


3.11ThinkForward shall use a scoring mechanism based on a weighting of NEET risk factors to produce a list of the Year 
10 and 11 pupils ranked according to their risk of becoming NEET. 


3.12All pupils in Year 10 and 11 will be surveyed at the start of the 2013/14 academic year to collect data to be used as 
a preliminary assessment of their mind set and employability skills and expectations, and a point of comparison for 
later analyses. 


3.13All pupils in Year 10 and 11 will be surveyed again at the end of the 2013/14 academic year. 
3.14 During the 2014/15 academic year, all pupils at the end of Year 11 will be surveyed for a third and final time. 


3.15Data from these surveys will be merged and brought together into a single file and linked to data from the National 
Pupil Database using Unique Pupil Numbers (UPNs). 


3.16 The final merged and linked data file will be anonymised so that pupils are only identified by their UPN. 
3.17The School/Academy Lead and Data Coordinator will be consulted during all these data gathering processes. 
3.18Participant and parental consent will be sought for this data collection exercise on an opt-out basis. 


3.19The School/Academy will receive a covering grant of £5000 in payment for any associated costs involved in this 
data collection. 


B) Random Selection of Schools 


2.1 Your school will be randomly allocated to receive ThinkForward or to become a Control group 
School/Academy. 


2.2 For the research, the Control group School / Academy is as important as the School / Academy receiving 
ThinkForward because without Control group Schools we will not be able to see whether ThinkForward has a causal 
impact. 


For Schools Selected to Receive ThinkForward 


C) Selection of Pupils to Receive ThinkForward 


3.1 Using the ThinkForward scoring mechanism, 60 pupils from both Years 10 and 11 will be identified as being at 
most risk of becoming NEET. 
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3.2 Participant and Parental consent would be sought from these 60 identified pupils in both Years 10 and 11 on an 
opt-out basis. 


3.3 Twenty of the identified and consenting pupils (in both Years 10 and 11) will be randomly selected to work with 
a ThinkForward Coach Twenty pupils will be randomly selected to become Control group pupils. 


3.4 Should any young people selected for the programme decline to participate, they will still be counted as 
participating in the trial, for the sake of fairness. However, the School/Academy and Coach may choose to 
engage other participants (not on the original list of at risk young people) to benefit from the programme instead. 


D) Further Data Collection Activities 


4.1 Those 40 pupils in both Years 10 and 11 that are randomly selected to take part in the trial (either in the 
treatment or in the control group) will be interviewed as they reach the end of Year 11. 


4.2 These pupils will be interviewed in small groups 

4.3 Pupils currently at the start of Year 11 will be interviewed towards the end of the 2013/14 academic year. 

4.4 Pupils currently at the start of Year 10 will be interviewed towards the end of the 2014/15 academic year. 

4.5 Year 10 and 11 Teachers and Year Tutors will also be interviewed individually at a similar time to the pupils. 
E) Enrolling young people on ThinkForward 


5.1 Once young people have been selected to work with a Coach, the relevant Head of Year and Form Tutors shall 
support the Coach to establish initial contact, which may be by arranging an initial meeting. 


5.2 During the initial meeting, the Coach will seek the young person’s agreement to be involved. The young person 
and Coach shall complete and agree the ThinkForward Enrolment and Consent form (See documentation pack). 
Consent is required that: 

= ThinkForward may share anonymised information with government departments. This information is treated 
in the strictest of confidence and used for research and statistics only 

= Participants may be asked to complete a feedback survey. 

= ThinkForward funders may contact the School/Academy or Employer to confirm successful outcomes 


Their parents/carer is also required to sign the consent form. 


5.3. The young person and Coach may also agree an informal contract (See documentation pack) which will set out 
= Young person expectations of the Coach 
= Coach expectations of the young person 


Both parties shall sign the contract after it has been personalised. The young person should keep a copy and 
share it with parents/carers. 


F) Assessing the young person’s needs and capabilities 


3.1 The Coach shall hold an initial 1-1 assessment with each young person, where they will discuss and record the 
young person’s goals, barriers and development needs (See documentation pack). 


3.2 The Coach shall work with the young people selected for ThinkForward to review the ThinkForward Personal 
Development Questionnaire which they completed during the identification process (see Step A, above). They 
will work through and discuss a report based on the results of the Questionnaire which assesses the young 
person’s mindset and skills with a view to employability: 


Mindset Skills 
e = Self Belief e People Skills 
e Positive Thinking e Teamwork 
e Aspiration e Effective Communication 
e Determination e Finding Solutions 
e —- Flexibility e Planning & Organising 
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e Appetite For Learning e Building a Positive Network 
e Managing Emotions 


e Understanding Emotions 


This Personal Development report will form the basis of the Coach’s initial work with the young person. For 
that reason, pupils who have not been selected for ThinkForward will not be able to access this report. 
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Designing an action plan 


4.1 The Coach shall, in consultation with School/Academy personnel, collaboratively build a ThinkForward Action 
Plan (See documentation pack) in a one off design meeting. The objective of the Action Plan is to structure a 
young person’s development and to provide a consistent point of reference as they work through the 
programme. 


4.2 The Action Plan, which shall be made available to the TP Operational Lead Manager and the School/Academy 
Lead, should include: 
= Long term goals (SMART) 
= Realities/barriers: an expression of the current situation 
= Short to medium term actions and opportunities 
= Development path (interventions planned, interventions completed) 
= Next steps and opportunities to review 


4.3 Future updates to the Action Plan shall be the responsibility of the Coach and should be driven either by a pre- 
agreed schedule or by a change in the young person’s goals or circumstances. 


Delivering against action plan 


5.1 During the Action Plan design / review meetings, the Coach and young person shall identify interventions which 
may meet the young person’s development needs. 


5.2 The Coach shall provide regular 1:1 coaching sessions with each young person, based on a solutions-focused 
approach. The Coach shall also lead group sessions in the School/Academy with ThinkForward pupils. 


5.3 The Coach may organise activities within the School/Academy but led by an external provider, or external 
interventions for individuals or groups where beneficial for young people, always informing School/Academy 
personnel as required. 


5.4 The Coach has access to a ThinkForward ‘Interventions Catalogue’ which maps the high quality provision in 
your local area. The Coach shall monitor the quality and fitness of these external interventions on your behalf. 


5.5 The Coach may act as a ‘NEET specialist’ within the school — developing policies, mentoring teachers and 
tutors, increasing awareness, and providing external challenge to the schools existing support services. 


5.6 The School/Academy’s External Visits Coordinator shall approve and risk-assess any offsite trips. Decisions 
about what interventions are suitable shall be entirely at the School/Academy’s discretion (see below for health 
& safety). 


5.7 The interventions agreed upon in the Action Plan may involve local and national employers. ThinkForward’s 
employer engagement model (see Schedule 1) sets out a likely path into employment for young people on the 
ThinkForward programme. 


5.8 This engagement model includes opportunities to participate in a business mentoring scheme, in which pupils 
are matched in ‘mentoring families’. Meetings may take place both in the School/Academy and at their 
workplace. Contact your Coach to register interest in participating in a scheme. 


Supporting success: achieving Key Performance Indicator Outcomes 


7.1 ThinkForward sets targets for success with the Coach based on a number of positive outcomes with young 
people (see Schedule 3). At school age, these outcomes hinge around 


= Improved behaviour at school 
= Improved attendance at school 
= Achievement of NQF Level 1 and Level 2 qualifications 


7.2 The ThinkForward Coach undertakes to work towards these goals with participants at your school. They shall 
on a termly basis give the School/Academy Lead, Head of Year and Form Tutor feedback on young people’s 
progress, feeding into School/Academy monitoring and assessment processes as required. 
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J) Transitioning to further education, training or employment 
8.1 The post-16 positive outcome targets of the ThinkForward programme hinge around: 
= Achieving NOF Level 3 qualifications 
= Entering further education to achieve an NOF Level 4 qualification 
= Entering and sustaining employment of more than 16 hours per week 


8.2 The ThinkForward Coach will help the School/Academy to broker access to further education institutions, and 
where possible make agreements with those institutions to the benefit of ThinkForward participants. 
ThinkForward aims to make similar agreements and access arrangements with Apprenticeship schemes. 


8.3 The Coach will guide and support ThinkForward participants through the process of applying for and succeeding 
in their future destination to ensure a positive outcome. 


8.4 When the young people in the ThinkForward initiative leave the School/Academy and move on to post-16 
education, employment or training, the Coach shall record and report young people’s destinations to the 
School/Academy. 


8.5 The School/Academy Lead, Head of Year and Form Tutor shall be consulted on what information can be shared 
with any other agencies picking up support, in line with the Data Protection Act and the School/Academy’s 
policies. 


8.6 Where the Coach looses contact with a young person, the Coach and ThinkForward will seek to re-establish 
contact as soon as possible and will consult the School/Academy Lead, Head of Year and Form Tutor for 
advice. 


Responsibilities of School/Academy 


A) Supporting the Coach and the programme 


1.1 The School/Academy Lead shall provide operational direction to the Coach via an induction to include: 
= Allocating a member of their senior leadership team as the ‘ThinkForward Lead’ who will support the Coach 
on a day to day basis and champion the ThinkForward initiative within the School/Academy 
Meeting between ThinkForward Lead and Coach to take place at least monthly 
Involving the Coach as appropriate in their senior/middle leadership teams 
Ensuring the Coach is integrated into the school systems for inclusion and pastoral support 
Contributing to the operational priorities and work plan for the Coach 
Agreeing the pattern of work and range of interventions to be offered within the School/Academy 


1.2 Tomorrow's People shall allocate a member of their personnel as the Operational Lead Manager, who will be 
the School/Academy’s key point of contact for the ThinkForward initiative. They shall retain accountability for 
the line management and work of the Coach by: 

= Setting the organisational strategy and service standards 

Managing the partnership with each School/Academy 

Proactively monitoring delivery and performance, including through feedback from each School/Academy 

Promoting individual learning and development, through regular supervision and appraisals 

Developing external partnerships and other ways of working which support effective practice 

Taking corrective action on any deviations from the strategy or standards and to promote continuous 
improvement 


1.3 In the event of any concerns about the work of the Coach, the School/Academy Lead shall report any concerns 
to the Operational Lead Manager. The Operational Lead Manager will respond in line with the Tomorrow's 
People complaints policy and may escalate it depending on the severity. 
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Providing accommodation and communications resources 


2.1 


2:2 


2.3 


The Coach will require a desk within the School/Academy with access outside of school hours where 
necessary. 


The Coach will require a place to meet young people confidentially. They will also need a lockable place to store 
confidential documents and portable media. 


The Coach will require access to the School/Academy IT network and information systems, and the 
School/Academy phone system. 


Allowing ThinkForward to access pupil data 


3.1 


3.2 


3:3 


3.4 


3.5 


Data on at-risk of NEET indicators shall be collated by the Coach with help from School/Academy personnel, 
including: 

= End of Key Stage 2 and 3 national curriculum levels and levels of progress achieved 

= Attainment in formative assessments from the previous year 

= Attendance for previous year and year-to-date 

= Exclusion record for previous year and year-to-date 

= Evidence of learning difficulties/disabilities 

= Evidence of belonging to a vulnerable group, such as living in a workless household, young offender, young 

person in care or young parent, CAF statement. 


This data will be used to populate a NEET ‘scoring mechanism’ (see Schedule 1) which will help ThinkForward 
to identify the most appropriate caseloads. 


On an ongoing basis, the School/Academy Lead shall provide the Coach with access to the data held by the 
School/Academy in order to evidence the Key Performance Indicator Outcomes in Schedule 6. This data could 
include: 

= Attendance records and nature of absence 

= Behaviour and exclusion records 

= Qualification/attainment records and predicted grades 

The School/Academy Data Coordinator and other school personnel should assist and support the Coach in 

collating the data. Some of this data will be needed for the whole year groups rather than just those working 

with the ThinkForward Coach. 


ThinkForward may occasionally obtain a Case Study in a pre-approved template from the Coach (see 
documentation pack). Where this Case Study concerns the School/Academy, the Coach will seek the school’s 
approval before promulgating. 


The Coach and a member of staff from the School/Academy shall make themselves available for yearly 
interviews with the independent evaluator to discuss the progress of the school. The ThinkForward Lead shall 
also assist the evaluator to organise a yearly focus group with ThinkForward young people. 


All data shared with Tomorrow's People will be held in compliance with the Data Protection Act 1998 and as set 
out in clause 14.1. 


Releasing pupils during core school hours 


7.1 


7.2 


7.3 


Evidence for ThinkForward proves that the work of the Coach has a net benefit to young people in 
Schools/Academies even given the necessity for it to be carried out during the school day. Furthermore, this 
work may prevent loss of lesson time due to permanent or temporary exclusions in the future. 


The size and nature of the caseloads makes it necessary to do some of this work beyond lunchtimes and 
after school. The School/Academy Lead shall agree with the School/Academy when during the day coaching 
sessions can take place. 


The Coach will regularly review the young person’s progress and coordinate the input of other 
School/Academy personnel through the School/Academy’s existing inclusion meetings or highlighting panels 
(where they exist). 
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Other Functions 


A) HR and safeguarding 

9.1 The Operational Lead Manager / Coach shall consult the School/Academy Lead when agreeing the Coach’s 
annual leave. Leave should only be taken in term time with the explicit agreement of the Operational Lead 
Manager. 


9.2 The Operational Lead Manager / Coach shall inform the School/Academy Lead when the Coach is off sick. In 
the event of long-term absence, the Operational Lead Manager will arrange cover if necessary. 


9.3 The School/Academy Lead shall support the induction of the Coach into the School/Academy, covering: 


= Introduction to key personnel 


School/Academy IT Systems 
= School/Academy Child Protection policies and procedures 
= School/Academy Health & Safety policies and procedures, 


9.4 The Coach shall report all safeguarding causes for concern in line with the School/Academy Safeguarding 
Policy. 


9.5 A) The Coach shall deliver the programme in line with the School/Academy health and safety policies during 
term time; the External Visits Co-ordinator or equivalent member of staff shall make themselves available for 


consultation on this. 


B) Tomorrow’s People has insurance and Health & Safety processes which allow Coaches to take 
responsibility for activities during the holidays. Any further checks outside of school time required by the 
School/Academy can be accommodated. 


9.6 ThinkForward recognises that respecting young people’s confidentiality is a fundamental requirement for 
maintaining trust, and seeks to: 


= Balance the rights of young people with the duty to safeguard their wellbeing 


= Recognise that sharing confidential information may only be authorised by young people’s consent or the 
law 


= Support the sharing of confidential information without consent when (i) there is evidence or reasonable 
cause to believe that the child is suffering or is at risk of suffering significant harm; or (ii) to prevent 
serious crime, including through the prevention, detection and prosecution of serious crime 


9.7 The Operational Lead Manager shall maintain management oversight of the Coach’s caseload. They shall dip 
sample a representative proportion of the Coaches records to ensure they are following the ThinkForward 
methodology effectively and providing high-quality support. 


9.8 The Operational Lead Manager shall carry out termly review meetings with the School/Academy Lead. The 
School/Academy shall ensure that the appropriate people within the School/Academy attend the review 
meetings (either in person or by conference call). The School/Academy shall provide such operational and 
other information as we may reasonably request in advance of such meetings and answer any questions as 
we may reasonably ask at those meetings. 


9.9 The School/Academy Lead shall be consulted on the School/Academy’s feedback on the Coach’s performance 
during their appraisal process. 


B) IT and record-keeping 


10.1The Coach shall record young people’s progress in the ThinkForward management information system. The 
School/Academy Lead may seek access to this information when required. 
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10.2The School/Academy shall promptly make available to Tomorrow's People and the Private Equity Foundation 
such other information as we may reasonably ask for. 


10.3The School/Academy Lead shall enable the Coach to access their hardware and software on the 
School/Academy site and the School/Academy Data Coordinator shall provide access to SIMS and other 
School/Academy systems. This access shall be solely used to support young people and the development of 
the ThinkForward initiative. 


10.4The School/Academy will, where evidence suggests outcomes have been achieved, assist the Coach in 
acknowledging Key Performance Indicator outcomes as listed in Schedule 6. 


C) Finance 


11.1 Any expenditure incurred by the School/Academy that they wish to claim against the ThinkForward budget 
must be agreed in advance with the Operational Lead Manager. 


D) Monitoring and evaluation 


12.1 The School/Academy Lead and other School/Academy personnel shall take part in an annual review meeting 
and external evaluation. 


12.2 The School/Academy Lead shall feed into the development of the ThinkForward initiative through 
representation (directly or through a School/Academy’s representative) at the ThinkForward Schools 
Partnership Forum. 


E) Fundraising and marketing 
13.1 The ThinkForward Initiative is the intellectual property of the Private Equity Foundation (PEF). 


13.2 The School/Academy Lead and other School/Academy personnel shall co-operate with and endeavour to 
give such assistance to Tomorrow’s People and PEF as we may reasonably request on reasonable notice 
from time to time in connection with promotional, communications and stakeholder engagement activities 
organised by us. 


13.3 Such activities may include, but are not limited to, the publication of articles or promotional material, up to 
three visits per year and appearance and participation by you at up to three events per year for the 
ThinkForward members, supporters and sponsors. 


13.4 At our reasonable request, the School/Academy Lead and Tomorrow's People shall discuss and agree the 
text of a joint press release to be issued following the entry into of this Memorandum of Understanding and in 
line with the PEF and TP ThinkForward Agreement. 


13.5 The School/Academy shall include ThinkForward’s logo with reasonable prominence and refer to your 
support from, and relationship with, Tomorrow's People and PEF in your main promotional literature 
(including in your annual report) and on your website using an agreed wording. 


13.6 The School/Academy shall not otherwise make any public statement (including on your website) which refers 
to Tomorrow's People or PEF, without first agreeing the form of wording with Tomorrow's People and PEF 
(and they shall not unreasonably withhold or delay our consent). 


F) Confidential Information 


14.1 Tomorrow's People shall keep all confidential information relating to young people and the School/Academy 
strictly confidential and shall use it solely in connection with the ThinkForward programme. Tomorrow’s 
People shall not at any time disclose, without the School/Academy’s prior written consent, any confidential 
information, save as required by law, regulation or any governmental or competent regulatory authority to 
which you are subject or where the information is publicly available or where the confidential information is 
shared with professional advisers. 


14.2 The School/Academy shall keep all confidential information relating to the ThinkForward Initiative strictly 
confidential. This includes: (i) all information relating to TP and PEF, and their respective members, donors, 
directors, officers and employees and their respective assets and affairs: (ii) the content of this Memorandum 
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of Understanding and any related documents or agreements; (iii) all information about the ThinkForward case 
management process; provided that any information which at the time of disclosure to you is in the public 
domain or after its disclosure to you comes into the public domain for any reason except your failure to 
comply with this Memorandum of Understanding or any other confidentiality obligations shall not constitute 
Confidential Information. 


14.3 The obligations of confidentiality set out in clause 14.1 shall not apply to any information shared between 
Tomorrow’s People and PEF. 


G) Intellectual Property 

15.1 PEF shall own all Intellectual Property Rights relating to ThinkForward. 
H) Duration 

16.1 This agreement is valid for two years from 

I) Changes to the Memorandum of Understanding 


17.1 Any changes to the Memorandum of Understanding shall only be effective once confirmed by both of us in 
writing. 


J) Non-legally binding 


18.1 The Memorandum of Understanding is not a legally binding document, save as set out in class [13.1], [14] 
and [15.1]. 


Signed for and on behalf of: 


Tomorrow’s People Trust Limited: 


Name: Date: 
School/Academy 
Name: Date: 
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Appendix 5: Deriving the percentage unauthorised absence 
secondary outcome variables 


Data on unauthorised absences for 2012/13, 2013/14 and 2014/15 were obtained from the National Pupil 
Database (NPD). . 


The unauthorised absences data for 2012/13 represented the baseline. The unauthorised absences data for 
2013/14 (cohorts 1 and 2) and 2014/15 (cohort 2 only) represented the outcomes. 


The NPD collects details on the number of half sessions a pupil is required to attend for each term of the 
school year. Alongside this, the NPD collects details on the number of half sessions missed due to 
unauthorised absences. 


The unauthorised absence measures all relate to the spring and summer terms in 2012/13, 2013/14 and 
2014/15. 


From 2012/13, the NPD provides details on the number of half sessions and unauthorised absences in three 
terms - spring, summer and a summer 6" term - see list of variables below: 


e SessionsPossible_Spring_ablyy]: Number of spring term half sessions possible 
e SessionsPossible_Summer_ablyy] Number of summer term half sessions possible 
e SessionsPossible_Summeréth_ab[yy] Number of summer 6" term half sessions possible 


The total number of sessions was calculated by summing these three variables for each of the academic 
years. 


e UnauthorisedAbsence_Spring_ablyy]: Number of spring term half sessions missed 
e UnauthorisedAbsence_Summer_ab[yy] Number of summer term half sessions missed 
e UnauthorisedAbsence_Summer6th_ab[yy] Number of summer 6" term half sessions missed 


Similarly, the total number of sessions missed due to unauthorised absence was calculated by summing 
these three variables for each of the academic years. 


To standardise the unauthorised absence outcome variables, they were converted into a percentage using 
the following formula: 


pea TT ec number of half sessions missed due to unauthorised absences (spring, & summer serms)) 
OGOSENCES = Total number of half sessions possible (spring, & summer terms) 


For cohort 1 (who completed Y11 in 2013/14) the outcome related to the spring and summer term in the 
2013/14 academic year. For cohort 2 (who completed Y11 in 2014/15) the outcomes relate to the spring and 
summer term in the 2013/14 and 2014/15 academic years. 


As might be anticipated, the resulting distribution for the %absences outcome variables displayed a marked 
positive skew at all three time points. For cohort 1, this skew was further exacerbated with a particularly high 
outlying case that was recorded as being unauthorised absent for 219 out or a possible 234 sessions (94%) 
in 2013. After exploring a number of possible transformations to reduce the positive skew”, it was decided 
to derive a series of binary outcome variables to supplement the original scale (but strongly skewed) 


24 KN square root and reciprocal square root transformation was examined along with excluding very high outlying cases. However, a 
reduced but still problematic positive skew remained - which is why we proceeded to create a further three simplified binary outcome 
variables where the positive skew was removed completely. 
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measures. This means that in addition to the %absence outcome variables in 2013, 2014 and 2015, three 
further outcome variables were created that measured the probability of any recorded unauthorised absence 
in the three years. This was done by creating a binary outcome variable that distinguished between 
participants recorded as having one or more sessions as unauthorised absent (=1) or having no sessions 
recorded as unauthorised absent (=0). These binary outcomes measure will be used to examine the 
probability of intervention and control group participants recorded as having any unauthorised absence in 
spring and summer terms in 2012/13 (prior to baseline), 2013/14 and 2014/15. This completely removes 
the positive skew from the outcomes but also results in a dramatic reduction in the fine detail provided by the 
original %absences scale measures. However, the strong positive skew observed within the original 
%absence variables means that increased caution is needed when interpreting model coefficients estimating 
the ‘impact’ of ThinkForward on unauthorised absences. Our approach is to use both the original (detailed 
but skewed) measured and supplement these with the simplified binary versions, and compared the findings 
of both approaches. 


Table 1 presents statistical summarises of the variables used to derive the %absences outcome variables, 
the derived %absence outcome variables and their simplified binary versions for unauthorised absences in 
the spring and summer terms in 2012/13, 2013/14 and 2014/15. 
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Table 1: Unauthorised Absences in 2012/13 (prior to baseline / randomisation), 2013/14 & 2014/15 


Scale outcomes: % of half sessions missed due to unauthorised absence in spring & summer terms 


Spring & Summer Terms 2012/13 


Spring & Summer Terms 2013/14 


Cohort 1 when cohort 1 were in Y10 when cohort 1 were in Y11 
n No of half No of % of half n Number of | Number’ of) %_ of half 
sessions unauthorised sessions half sessions | unauthorised | sessions 
absences unauthorised absences absent 
absent 

mean (sd) mean (sd) mean (sd) mean (sd) mean (sd) mean (sd) 
Intervention group 40 234 (0.6) 10.0 (16.7) 4.3% (7.1) 40 165.0 (1.0) 5.7 (13.8) 3.5% (8.4) 
Pupil level control group 75 229 (29.8) 11.9 (27.9) 5.2% (11.9) 75 165.2 (1.4) 5.1 (10.1) 3.1% (6.1) 
School level control group 66 232 (2.8) 6.3 (11.0) 2.7% (4.7) 66 161.2 (1.0) 5.9 (15.0) 3.7% (9.4) 
Total Sample 181 230.8 (19.3) 9.4 (20.8) 4.1% (8.9) 181 163.7 (2.2) 5.5 (12.8) 3.4% (7.9) 
Mean Difference (cohens d effect size) 
Intervention - Pupil level control - - - -0.9 (-0.08) - - - +0.4 (+0.05) 
Intervention - School level control - - - +1.6 (+0.27) - - - -0.2 (-0.03) 


Simplified binary outcomes: — whether missed ANY half session due to unauthorised absence 


Cohort 1 2012/13 | 2013/14 
n number with 1+ n number with 1+ 
half session half session 
recorded as recorded as 
‘unauth abs' ‘unauth abs' 
Intervention group 40 28 (70%) 40 19 (48%) 
Pupil level control group 75 55 (73%) 75 37 (49%) 
School level control group 66 35 (53%) 66 28 (42%) 
Total / combined control 141 90 (64%) 141 65 (46%) 
Total Sample 181 118 (65%) 181 97 (46%) 
Odds-Ratio Differences 
Intervention : pupil level control - 0.85 - 0.93 
Intervention : school level control - 2.07 - 1.23 
Intervention : combined control - 1.32 - 1.06 
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Table 1 continued: 
Scale outcomes: 


Cohort 2 


n Number of half Number of % of half 
sessions absences sessions absent 
mean (sd) mean (sd) mean (sd) 

Intervention group 38 230 (24.0) 9.1 (9.1) 4.0% (3.9) 
Pupil level control group 55 234 (0.5) 9.7 (12.6) 4.1% (5.4) 
School level control group 69 230 (2.9) 6.7 (9.5) 2.9% (4.1) 
Total 162 231 (11.8) 8.2 (10.6) 3.6% (4.6) 
Mean Difference (cohens d effect size) 

Intervention - Pupil level control - - - -0.2 (-0.03) 
Intervention - School level control - - - +1.1 (+0.28) 


Cohort 2 


Spring & Summer Terms 2013/14 


when cohort 2 were in Y10 


% of half sessions missed due to unauthorised absence in spring & summer terms 


Spring & Summer Terms 2012/13 
when cohort 2 were in Y9 


Spring & Summer Terms 2013/14 
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% unauthorised Absences in 2012/13 (prior to baseline / randomisation), 2013/14 & 2014/15 


when cohort 2 were in Y11 


n Number of Number of % of sessions n Number of | Number of |% of sessions 

absences absences absent absences absences absent 

mean (sd) mean (sd) mean (sd) mean (sd) mean (sd) mean (sd) 
Intervention group 38 234 (19.4) 10.7 (21.1) 4.8% (9.4) 35 162 (25.6) 9.7 (22.4) 5.8% (13.3) 
Pupil level control group 55 233 (12.8) 8.8 (31.0) 3.8% (13.3) 54 164 (10.9) 5.4 (15.8) 3.3% (9.6) 
School level control group 69 223 (29.3) 8.2 (14.0) 3.8% (6.4) 65 165 (9.9) 5.0 (12.8) 3.0% (7.7) 
Total 162 229.0 (23.0) 9.0 (22.6) 4.0% (9.9) 154 164.1 (15.1) 6.2 (16.5) 3.7% (9.9) 
Mean Difference (cohens d effect 
size) 
Intervention - Pupil level control - - - +1.0 (+0.08) - - - +2.5 (+0.22) 
Intervention - School level control - - - +1.0 (+0.13) - - - +2.8 (+0.28) 
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Simplified binary outcomes: 


whether missed ANY half session due to unauthorised absence 


Cohort 2 YAO 7A} 2013/14 2014/15 
n No. with 1+ half n No. with 1+ half n No. with 1+ half 
session recorded as session recorded session recorded 
‘unauthorised absent’ as ‘unauthorised as ‘unauthorised 
(%) absent’ (%) absent’ (%) 
Intervention group 38 33 (87%) 38 29 (76%) 35 18 (51%) 
Pupil level control group 55 40 (73%) 55 31 (56%) 54 20 (37%) 
School level control group 69 41 (69%) 69 47 (68%) 65 30 (46%) 
Total / combined control 124 81 (65%) 124 78 (63%) 119 50 (42%) 
Total Sample 162 114 (70%) 162 107 (66%) 154 68 (44%) 
Odds-Ratio Differences 
Intervention : pupil level control - 2.48 - 2.49 - 1.80 
Intervention : school level control - 4.51 - 1.51 - 1.24 
Intervention : combined control - 3.50 - 1.90 - 1.46 
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Appendix 6: Deriving the ThinkForward ‘mind set’ scales 


As part of the ThinkForward coaching programme, coaches are required to collect data on the young 
people that they are working with using the ThinkForward questionnaire (ThinkForward Manual, 2013). 
The questionnaire collects details on eight 'mind sets' and six employability constructs as set out in 
Table 1 below. 


Table 1: ThinkForward ‘mind set' and employability constructs 

Mind Set Construct Alpha Employability Construct Alpha 
Self-Belief 0.82 Effective Communication 0.70 
Positive Thinking 0.74 People Skills 0.81 
Aspiration 0.83 Teamwork 0.86 
Determination 0.83 Finding Solutions 0.77 
Flexibility 0.73 Building a Positive Network 0.73 
Appetite for Learning 0.71 Planning & Organising 0.83 
Understanding Emotion 0.82 

Managing Emotion 0.85 


The Cronbach Alpha figures are taken from the ThinkForward manual (2013, page 9) and measure 
the internal consistency of the questionnaire items used to construct each of the 14 constructs. Each 
scale was constructed from responses to five Likert scale items that asked participants to indicate a 
level of agreement to each of the five statements. 


Prior to randomisation, ThinkForward selected two of the mind set constructs for inclusion within the 
secondary outcomes for the trial. The two constructs that they considered were the most likely to 
show a positive impact from the ThinkForward programme within the trial period were the ‘aspiration’ 
and ‘determination’ mind sets. 


The aspiration mind set scale was constructed from responses to the following statements: 


Aspiration mind set 
e | have clear goals that | want to accomplish 
| set myself goals every day. 
| know what | want to achieve in life. 
| Know what | would like to do when | am older. 
| have no idea what | want to do with my life.(R) 


Participants were asked to select whether they strongly disagreed, disagreed, were unsure, agreed or 
strongly agreed with each of the five statements. Responses were then brought together into the 
‘Aspiration’ mind set. The first four statements were coded from 1 (strongly disagree) up to 5 (strongly 
agree) whilst the final statement was reverse coded from 5 (strongly disagree) up to 1 (strongly 
agree). The responses were then added together to compute an ‘aspiration’ scale that ranged 
between 5 and 25 with higher values representing higher aspiration. 


A similar approach was taken for the determination mind set using the five Likert statements listed 
below. 


Determination mind set 
e =| finish whatever | start. 
Setbacks never stop me. 
| make sure | achieve my goals. 
| enjoy working hard. 
| give up when things become difficult.(R) 


The distributions for these mind sets at baseline, survey 2 and survey 3 are shown in Table 2.4.5 in 
section 2.4.5 of the main report. 
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The internal consistency of the aspiration and determination scales were observed to be weaker within 
our baseline sample for both the aspiration (alpha=0.76) and determination (alpha=0.65)*°. This 
might be a reflection of how the pupil samples within this pilot trial were not drawn to be representative 
of the wider pupil population. 


These two derived secondary outcome scales can be seen as the area perceived to be the most likely 
to display a positive impact from the ThinkForward programme.  ThinkForward selected them in 
September 2013 prior to randomisation. | Whilst the choice of selection is not rooted within a 
developed logic model, the reason given by ThinkForward for their inclusion was: 


"we feel ‘Aspiration’ and ‘Determination’, as two of the statistically strongest, would be 
interesting and helpful, and make for a holistic evaluation." 


A few methodological notes about these two ThinkForward mind set scales needs to be made. First, 
these measures were designed by ThinkForward and are used by ThinkForward coaches. This 
means that intervention group pupils will have met these items on more occasions than pupil-level or 
school-level control groups. This also means that these scales will be more open to an inherent to 
treatment bias. 


In addition to these two important methodological caveats about these particular measures, it is worth 
restating four issues regarding the data collection for the secondary outcome measures. First, it 
should be restated that the data collection for both the impact and process elements of this pilot 
evaluation were facilitated by the ThinkForward coach. The coach gave out and collected the 
questionnaires and assisted in arranging interviews with pupils and staff. | Second, the transfer of 
questionnaires from the four London schools to us for processing and analyses in Sheffield took 
notably more time than was expected. Third, when attempting to collect questionnaires following the 
final survey we were told that a number of them had gone to ‘ThinkForward' head office but would 
eventually be sent on to Sheffield. Fourth, in the final survey, we found a number of instances where 
we had two completed questionnaires for an individual student®®. 


Some of these issues may reflect some naivety around the methodological requirements of an RCT 
from ThinkForward?’ but any future trial would need to design out these apparent conflicts of interest 
within the data collection process. This might be done through a more direct data collection 
approach” and the use of invigilators. 


Taken together, the methodological caveats and data collection issues that relate to the ThinkForward 
mind set scales serve to emphasize the need for great caution when interpreting impact or ‘evidence 


25 These Cronbach Alpha statistics were calculated for the combined Y10/11 samples and cohorts. 
i.e. n=288 (149 from cohort 1 +139 from cohort 2) for the aspiration scale; n=295 (152 from cohort 1 
+143 from cohort 2) for the aspiration scale 


26 When this was found we excluded the pupil from the analysis (i.e. we set all of their responses to be 
missing for the final survey). This, however, did not ensure that some of the other questionnaires 
were not second attempts 


27 For example, we were informed by one of the coaches that they had requested some pupils to 'redo' 
the questionnaire because they ‘had not taken it seriously’. 


28 For example, an online approach might be preferable. Once a baseline survey is collected, future 
surveys could be directed at individual students with the use of follow-up reminders from the school 
(e.g. within an IT classroom). Invigilators could be randomly assigned to schools to oversee. This 
would design-out the potential subjective or objective conflict of interest bias we experienced within 
this pilot. This would also bring benefits in terms of linking responses across the surveys - a process 
that was more time consuming with physical paper questionnaires. 
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of promise’ findings relating to them. More so than any of the other outcomes, the ThinkForward mind 
set scales are set up to show a positive impact. At the same time, more so than any other outcomes, 
the ThinkForward mind set scales open to bias in overstating the impact of the 
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Appendix 7: Calculating Hedges g Effect Size from model 
coefficients 


KS4/GCSE attainment for cohort 1 used as an example 


An effect size is a statistical estimate of the strength of a phenomenon in standardised units. In the 
context of this research, the effect size provides an indication of the difference between the 
intervention and control groups for the outcome measures. Whilst the model coefficients also provide 
an indication of this, the effect size standardises these coefficients so that they can be compared 
directly with each other and across other research studies. Without standardisation, the size of 
coefficient is dependent on the scale and units of the outcome measure and so it is not possible to 
compare these directly. 


As specified by the EEF, the effect size calculated in this report is Hedges g. This is a similar effect 
size statistic to Cohen’s d but includes a correction factor to adjust for bias when dealing with small 
samples. The tables that follow provide some summary details on the primary outcome measure 
(KS4 attainment) and how the Hedge's g estimates were calculated. 


Primary Outcome: Mean KS4/ GCSE points per entry 


Mean 34.4 35.4 34.6 
Standard deviation 8.99 6.76 8.54 
n= 141 40 181 


Pooled standard deviation* 8.56 


Mean 36.9 35.4 36.4 
Standard deviation 10.13 6.76 9.10 
n= 75 40 115 


Pooled standard deviation* 9.11 


The pooled standard deviation is calculated using the following formula: 


[e — 1)s? + (n, - 1)s? 
s= | 


ny +n,—2 


Where s is the pooled standard deviation, Si is the standard deviation for the intervention group and sz 
is the standard deviation for the control group. ni is the number of participants in the intervention 
group and nz is the number of participants in the control group. 


Referring to Table 3.3.1a in the main report, the following intervention coefficients are reported for the 
main effects models for the KS4/GCSE attainment outcome for cohort 1 of the pilot trial. The 95% 
confidence intervals were provided through the STATA regression command with standard errors 
adjusted for clustering within schools. 


Intervention / 95% Confidence Intervals 


Treatment Coefficient Lower Upper 
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All 4 schools +0.6 (0.56) -1.23 +2.36 
2 intervention schools only +0.2 (0.60) -7.45 +7.85 


Dividing the coefficient estimates (and the upper and lower 95% confidence intervals) by the 
appropriate pooled standard deviations converts them to Cohen's d effect size statistics. 


For example, for the coefficient estimated from the sample of 2 intervention schools. 


Cohen's d = B1 / (pooled sd) = 0.20 / (9.11) = 0.022 ~ +0.02 
Converting coefficients & 95% 95% Confidence Intervals 
Cls into Cohen's d effect size Cohen's d Lower Upper 
Seusies 
All 4 schools +0.07 -0.14 +0.28 
2 intervention schools only +0.02 -0.82 +0.86 


The Hedges g effect size is calculated by multiplying the Cohen's d effect size statistic by the following correction 
factor: 


3 


Hedges g correction factor = 1 — ———————— 
ae f 4(n, +n2) -9 


This results in two correction factors for the four-school and two-school research designs - shown in the following 
table: 


Converting coefficients & Hedges g 95% Confidence Intervals 
95% Cls into Hedges g effect Correction Lower Upper 
S4E5 Factor 

All 4 schools 0.996 +0.07 -0.14 +0.27 
2 intervention schools only 0.993 +0.02 -0.81 +0.86 


These are the (stage 3) effect size estimates reported in Table 3.3.1a of the main report. The units are standard 
deviations for the KS4 / GCSE attainment outcome variable for cohort 1 (who completed KS4 in Y11 in summer 
2014). 
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Appendix 8: Calculating odds-ratios from model 
coefficients 


Simplified (binary) version of unauthorised absences outcome 
Cohort 1 (in Y11 during 2013/14) 
Probability (1+ instance of unauthorised absences in spring & summer terms, 2013/14) 


The simplified binary version of the unauthorised absence outcomes was created to 
remove a strong positive skew present within the percentage scale version of the 
outcome (see Appendix 5). This skew might lead to bias in estimating the impact of 
ThinkForward on unauthorised absences. The more detailed (but skewed) 
percentage scale outcomes were modelled using difference in difference linear 
regression and coefficients converted into Hedges g effect size statistics (See 
Appendix 7). The simplified binary (but not skewed) outcomes were modelled using 
difference in difference logistic regression. The interpretation of logistic regression 
coefficients relates to odds-ratios. 


Difference in difference logistic regression was used to model the simplified binary 
unauthorised absences outcome variables. These models involve a logit link function 
that converts the probabilities (of 1+ absence) into log-odds. This means that the 
model coefficients within the models are not easily directly readable. To aid 
interpretation, the exponential of the coefficients can be taken. This converts the 
coefficients into odds-ratios. This is the relative odds of one group compared with 
another. In the case of this trial, the odds of absences for pupils in the intervention 
group compared with the odds of absence for pupils in the control group. 


e Odds ratios = 1.0 indicate that the intervention and control group had similar 
odds of unauthorised absences. 

e Odds ratios < 1.0 indicate that the intervention group were less likely to have a 
recorded instance of unauthorised absences compared with the control group. 

e Odds ratios >1.0 indicate that the intervention group were more likely to have 
a recorded instance of unauthorised absences compared with the control 
group. 

Referring to Table 3.3.2 in section 3.3.2 of the main report. 


For cohort 1, the (logit) coefficient for the intervention*time interaction dummy 
variable and the 95% confidence intervals obtained from STATA are shown below. 
This relates to unauthorised absences in spring / Summer 2013/14 and the stage 2 
difference in difference logistic regression models. 


Cohort 1, All 4 schools Coefficient (Logit) 
Unauthorised absences in (95% Cl) 
spring/summer 2013/14 

Stage 2 -0.37 (-2.56; +1.82) 
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By taking the exponential of the coefficient (and confidence intervals), the logit can 
be converted into odds-ratio effect size statistics - as shown below and in Table 
3.3.2a of the main report. 


Cohort 1, All 4 schools Exp(fs) - Odds Ratio 
Unauthorised absences in 

spring/summer 2013/14 

Stage 2 0.69 (0.08; 6.17) 


Cohort 1, the intervention group are 69% as likely to have 1+ instance of 
unauthorised absence in spring/summer 2013/14 compared with the control group. 
The 95% confidence intervals range from below 1 (less than a tenth as likely) to 
above 1 (over six times as likely) - in other words, the coefficient is not statistically 
significant. 
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Appendix 9: Technical appendix 


A spreadsheet containing the full technical appendix is available on request. This include full model 
details (for GCSE attainment and unauthorized absences). 


If you would like access to the technical appendix, please email info@eefoundation.org.uk 
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